Method and system for providing remote robotic control

ABSTRACT

A virtual pointer object is displayed within a virtualized environment corresponding to a physical environment currently surrounding a robot. The virtualized environment is generated and updated in accordance with streaming environment data received from sensors collocated with the robot. First user input is detected via a haptic-enabled input device that causes the virtual pointer object to move along a movement path in the virtualized environment, where the movement path is constrained by simulated surfaces in the virtualized environment. Haptic feedback is generated via the haptic-enabled input device in accordance with simulated material and/or structural characteristics of the movement path. The virtualized environment is modified at the locations of marking inputs along the movement path and affects path planning for the robot within the first physical environment.

TECHNICAL FIELD

This relates generally to robotic control, including but not limited toproviding mixed-initiative remote robotic control.

BACKGROUND

Today, robots are being used to perform or assist in many tasks thatpreviously were performed by human beings. In some instances, robots arebetter suited for certain tasks than humans due to their precision inmovement and control and their highly customizable designs and operationmodes. Some suitable tasks for remotely controlled robots include tasksin hazardous environments, such as natural disaster areas, toxicenvironment, virus or radiation contaminated environment, etc. In someexamples, tasks that are highly repetitive and tedious, and/orphysically strenuous tasks, such as performing household chores,organizing warehouse inventories, routine care and assistance to thedisabled and/or elderly, etc., can benefit greatly from roboticassistance.

Although teleoperation and complete automation are two modes of roboticcontrol that are widely used today, the kinds of tasks that can beperformed using these two modes of control are limited. For example,with teleoperation, a human operator precisely controls every move ofthe robot by performing a corresponding control movement in real-timewhile watching a streaming video of the scene in which the robot isoperating. The teleoperation mode of control requires a highly skilledhuman operator to be fully engaged (e.g., physically and mentally) inreal-time throughout performance of a task, which severely limits itsusefulness and applications (e.g., due to the burdens and requirementsplaced on the human operators). The full automation mode of controlrequires artificial intelligence to be built into the robot controlprograms, which cannot adapt to highly varied operation scenarios andlimits its usefulness and applications.

In a mixed-initiative robotic control mode, a human operator provideshigh-level task-oriented instructions, and relies on the robot'spre-programming to determine the exact movement and operations to beperformed to achieve the high-level instructions. Mixed-initiativerobotic control alleviates some burdens and requirements placed on thehuman operators, and takes advantage of the robot's capabilities interms of precision, stamina, robustness, and relative immunity tohazards.

It is challenging to provide an intuitive and efficient human-machineinterface that facilitates remote mixed-initiative robotic control. Goodtechniques for supporting indirect and/or intermittent intervention in arobot's operation (e.g., navigation and/or object manipulation) inreal-time over a network are in great need.

SUMMARY

Accordingly, there is a great need for mixed-initiative robotic controltechniques where a human operator controlling the robot visualizes a 3Dscene as perceived by the robot in real-time and guides the robot'sactions on a high-level (as opposed to controlling the robot viateleoperation) using indirect and/or intermittent instructions andinputs. When providing remote mixed-initiative control to robots, amixed reality environment is utilized for the user to perceive the 3Dscene and provide guidance inputs. The mixed reality environment is acombination of a virtualized representation of the physical environmentas perceived by the robot as well as virtual markers and virtual objectsthat are placed into the virtualized representation of the physicalenvironment. This technique provides the user more freedom andinformation during the interaction with the physical environment via therobot, such as switching viewpoints in real-time, inspecting selectedportions of the environment, receiving augmentation of visualinformation (e.g., texture information, physical property information,motion and appearance history, etc.).

In this disclosure, embodiments of a haptic-enabled mixed reality systemthat provides haptic interfaces to interact with a virtualizedrepresentation of a physical environment and to provide remotehigh-level guidance to mobile robots in performing high-level tasks aredescribed. The system includes a local site with a mobile robot equippedwith Red Green Blue Depth (RGBD) sensors, and a remote site with a useroperating a haptic-enabled input/output (I/O) device. Athree-dimensional virtualized real-world scene is generated fromstreaming RGBD data using real-time dense mapping. The user can use thehaptic-enabled input device to “touch” and experience physicalcharacteristics of surfaces and objects within the scene, to mark thescene to indicate virtual routes, passages, obstacles, and/or targetsfor the robot in the scene, and to add virtual objects to the scene thatalter the scene's composition from the perspective of the robot andthereby indirectly influence the exact operation that is performed bythe robot in accordance with preprogrammed instructions. In someembodiments, the system performs physics simulation for a virtualobject's interactions with other objects in the scene, to help the userexperience the scene and more accurately determine where to place thevirtual object in the scene. In some embodiments, real-time userassisted object segmentation is performed. The user uses thehaptic-enabled input device to mark a location in the scene, and anobject is segmented from the scene by a region growing method from themarked location. In some embodiments, a new processing pipeline forhaptic rendering is used to handle proxy update. The proposed proxyupdate method with force shading that is reflective of friction andhaptic textures is more stable and more realistically handlesintersecting boundaries of different planes in the scene. In someembodiments, prediction of actual robot position is made in light ofexpected network latency, such that real-time operation and control ofthe robot over a latency network is more smooth (e.g., with less jerky,back-forth movement of the robot).

In one aspect, a method of providing mixed-initiative robotic control isperformed at a computing device having one or more processors andmemory, wherein the computing device is communicably coupled to a robotand is configured to generate a planned path for the robot in accordancewith a first set of preprogrammed path-planning instructions, and therobot is configured to navigate within a physical environment inaccordance with the planned path received from the computing device andlocally-stored path-execution instructions. The method includes:displaying a control user interface via a display generation componentcoupled to the computing device, including displaying a virtual pointerobject within a virtualized environment corresponding to a firstphysical environment currently surrounding the robot, wherein thevirtualized environment is generated and updated in accordance withstreaming environment data received from a first set of sensorscollocated with the robot; while displaying the virtual pointer objectin the virtualized environment, detecting first user input via ahaptic-enabled input device, including detecting a first movement inputand one or more marking inputs at one or more locations along a movementpath of the virtual pointer object in the virtualized environment; andin response to detecting the first user input: moving the virtualpointer object along a first movement path in the virtualizedenvironment in accordance with the first movement input, wherein thefirst movement path is constrained by one or more simulated surfaces inthe virtualized environment; generating haptic feedback via thehaptic-enabled input device in accordance with simulated material and/orstructural characteristics of the first movement path in the virtualizedenvironment; and modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment, wherein the modified virtualizedenvironment replaces the virtualized environment as environment basisfor path planning for the robot within the first physical environmentthat is performed by the computing device in accordance with the firstset of preprogrammed path-planning instructions.

In one aspect, a method of performing interactive object segmentationfrom streaming surfaces is performed at a computing device having one ormore processors and memory, wherein the computing device is communicablycoupled, via a network, to a set of sensors and a robot that arecollocated in a first physical environment. The method includes:receiving an environment data stream from the set of sensors, theenvironment data stream including correlated image and depth datacapturing the first physical environment; displaying, via a displaygeneration component, a virtualized representation of the first physicalenvironment, in a user interface region, wherein the virtualizedrepresentation of the first physical environment is generated andupdated in accordance with the environment data stream; while displayingthe virtualized representation of the first physical environment,receiving a marking input from a haptic-enabled input device, wherein aposition of the marking input in the virtualized representation of thefirst physical environment is determined in accordance with the markinginput and is constrained by the a first virtualized surface in thevirtualized representation of the first physical environment; and inresponse to receiving the marking input, performing object segmentationfrom the position of the marking input on the correlated image and depthdata, including, for a respective time-step in updating the virtualizedrepresentation of the first physical environment in accordance with theenvironment data stream: clustering pixels from a first position in theimage data that corresponds to the position of the marking input to forma first cluster; segmenting out a first labeled object based on thefirst cluster and the depth data corresponding to the first cluster;fusing the first labeled object and the depth data into a firstTruncated Signed Distance Function (TSDF) representation of the firstphysical environment corresponding to the respective time-step; andobtaining a first segmented point cloud from the first TSDFrepresentation fused with the first labeled object and the depth datacorresponding to the first cluster.

In an aspect, a method of providing mixed-initiative robotic control isperformed at a computing device having one or more processors andmemory, wherein the computing device is communicably coupled to a robotand is configured to generate a planned path for the robot in accordancewith a first set of preprogrammed path-planning instructions, and therobot is configured to navigate within a physical environment inaccordance with the planned path received from the computing device andlocally-stored path-execution instructions. The method includes:displaying a control user interface via a display generation componentcoupled to the computing device, including displaying a virtualizedenvironment corresponding to a first physical environment currentlysurrounding the robot, wherein the virtualized environment is generatedand updated in accordance with streaming environment data received froma first set of sensors collocated with the robot; while displaying thevirtualized environment, detecting a first user input inserting a firstvirtual object at a first location in the virtualized environment; inresponse to detecting the first user input, modifying the firstvirtualized environment in accordance with the insertion of the firstvirtual object at the first location, wherein the first virtual objectat the first location causes the robot to execute a first navigationpath in the physical environment that is generated in accordance withthe first set of pre-programmed path-planning instructions; whiledisplaying the first virtual object at the first location in thevirtualized environment and while the robot is executing the firstnavigation path in the physical environment, detecting a second userinput, including detecting a first movement input directed to the firstvirtual object via a haptic-enabled input device; and in response todetecting the second user input: moving the first virtual object along afirst movement path to a second location in the virtualized environmentin accordance with the first movement input, wherein the first movementpath is constrained by one or more simulated surfaces in the virtualizedenvironment, wherein the first virtual object at the second locationcauses the robot to execute a modified navigation path in the physicalenvironment that is generated in accordance with the first set ofpre-programmed path-planning instructions.

In accordance with some implementations, a computing system includes oneor more processors, memory, and one or more programs; the one or moreprograms are stored in the memory and configured to be executed by theone or more processors and the one or more programs include instructionsfor performing the operations of any of the methods described above. Inaccordance with some implementations, a non-transitory computer readablestorage medium has stored therein instructions which when executed by acomputing system with one or more processors, cause the computing systemto perform the operations of any of the methods described above. Inaccordance with some implementations, a computing system includes meansfor performing the operations of any of the methods described above.

Additional advantages of the disclosed systems and methods are describedthroughout this disclosure, and/or are apparent to a person skilled inthe art in light of the disclosure provided herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described implementations,reference should be made to the Description of Implementations below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is an exemplary mixed-initiative robotic control environment inaccordance with some implementations.

FIG. 2 is a block diagram illustrating an exemplary architecture thatincludes the mixed-initiative robotic control environment in accordancewith some implementations.

FIG. 3 illustrates an exemplary processing pipeline for real-time visualand haptic rendering on the remote site of the mixed-initiative roboticcontrol environment, in accordance with some implementations.

FIG. 4 illustrates a proxy update method using force shading, inaccordance with some embodiments.

FIG. 5 illustrates proxy update for friction, in accordance with someembodiments.

FIG. 6 illustrates an exemplary processing pipeline for interactiveregion growing 2D segmentation method, in accordance with someembodiments.

FIGS. 7A-7C illustrate various examples of haptic markings and virtualobjects to influence route planning and operation of the robot, inaccordance with some embodiments.

FIG. 8 is a flowchart of a method of providing mixed-initiative roboticcontrol, in accordance with some embodiments.

FIG. 9 is a flowchart of a method of performing interactive objectsegmentation from streaming surfaces, in accordance with someembodiments.

FIG. 10 is a flowchart of a method of providing mixed-initiative roboticcontrol, in accordance with some embodiments.

FIG. 11 is a block diagram illustrating an exemplary robot in accordancewith some embodiments.

FIG. 12 is a block diagram illustrating an exemplary server inaccordance with some implementations.

FIG. 13 is a block diagram illustrating an exemplary user-side device inaccordance with some implementations.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DESCRIPTION OF IMPLEMENTATIONS

This disclosure provides system and method for providingmixed-initiative robotic control in accordance with various embodiments.

Mixed-initiative robotic control has advantages over teleoperation andfully autonomous robotic control using preprogrammed instructions due toits flexibility and real-time adaptability to new situations withintermittent aid of a remotely located human operator. On the one hand,the robot is programmed to perform various low level or intermediatelevel tasks in accordance with environment input collected via on-boardsensors and/or through a network, on the other hand, a human user canintervene indirectly at suitable times by modifying the perceivedenvironment input, in accordance with actual real-time visual and hapticrendering of the scene based on streaming RGBD data received from thesensors collocated with robot. The haptic feedback provided to the uservia a haptic-enabled input device allows the user to explore, touch, andexperience the scene via a haptic cursor or to interact with the scenethrough simulated physical interactions between a user controlledvirtual object and the scene. The flexibility and adaptability ofhaptic-enabled mixed-initiative remote robotic control allows the systemto be used in many situations that are too tedious, strenuous, and/ordangerous for human user and/or situations that are too complex andrapidly changing for fully autonomous robots. In addition, the level ofuser intervention is adjustable based on an actual situation inreal-time, further improving the performance and adaptability of thesystem.

Below are some example scenarios in which the haptic-enabledmixed-initiative remote robotic control system and method can beutilized with advantage over conventional systems.

In one example, in taking care of patients in field hospitals in remoteepidemic disease areas, robots can assist in caring for patients thatare highly contagious. Simple tasks such as moving from patient topatient to check their vitals, delivering medication and water topatients at scheduled times, removing soiled linens, cleaning up trashon the floors or bedside tables, etc. Due to the varied conditions inthe field hospital rooms, artificial intelligence of autonomous robotsbased on machine learning and preprogramming may not work very well insuch situations (e.g., for lack of training samples similar to the fieldhospital rooms, and lack of sufficient amount of time for training andperfecting the robot's decision-making programs). At the same time,regular hospital staff and doctors are busy with their regular duties,and may not have the physical stamina and/or undivided attentionrequired to remotely control robots using the teleoperation controlmode. In such cases, mixed-initiative remote robotic control with hapticfeedback is particularly useful. It relies on the robot's existingprogramming for low level tasks that are well trained and programmed(e.g., navigating to an identified target along an obstacle-free path,planning and executing a path to avoid obstacles that have beenidentified in its field of view, and grabbing and putting down anidentified target object) and it can also benefit greatly from a humanuser's help in identifying which objects in the scene is a target, anobstacle, a passage, and/or the best route to approach a target and/oravoid possible obstacles.

In another example, for domestic robots that help with assisting elderlypeople and managing household chores, the same customized robots can beshipped and delivered to different households, and the preprogrammedoperation and navigation instructions may work well for standardizedtasks and room layouts, such as sweeping the floor, wiping countertops,taking out trash from the trash can, etc. However, when the rooms arelittered after a party, or when the household members randomly leavevarious types items around the house, or when there are children or petsmoving about in and out of the rooms, the limited image processing anddecision making capabilities afforded by the standard preprogrammedinstructions will not be adequate in properly characterizing andtreating these highly varied environment inputs. Furthermore, the sameun-customized robots shipped to different houses (e.g., houses withdifferent appearances and layouts, different degrees of organization andcleanliness requirements, and different sensitivity to accidental errorsand collisions between the robot and the environment), haptic-enabledmixed-initiative remote robotic control allows a human user to tailorthe amount and types of control and intervention provided to theun-customized robot, such that the robot performs customized tasks inits actual customized environment. In some embodiments, the humanguidance history is collected overtime, such that the un-customizedrobot may utilize the history to self-training through machine learningand other artificial intelligence training methods to become more andmore autonomous in each household.

In another example, in an industrial manufacturing setting, manual laboris largely replaced by computer-controlled machines and autonomousrobots. However, certain tasks remain manually driven, due to theinfrequency of the tasks and/or high sensitivities to machine errors andaccidents. For infrequent tasks, it is not financially feasible todesign and program the robot to specialized in performing those tasksand it is difficult to know when those specially designed robots shouldbe deployed due to the unpredictability of when the needs for each taskwould arise. Furthermore, with expensive equipment, dangerous chemicals,and/or highly sensitive machinery or measurements in the operatingenvironment, fully autonomous robots are too unreliable to be deployedand utilized without human supervision. In such cases, haptic-enabledmixed-initiative remote robotic control utilizes a human user to providehigh-level guidance in real-time and to structure and guide thelow-level navigation and operation capabilities of the robot, such thatthe robot operates with sufficient guidelines and ample margin formachine errors when performing these highly infrequent and/or variedtasks in highly sensitive environments.

There are innumerable example scenarios where haptic-enabledmixed-initiative remote robotic control would outperform conventionalteleoperation, or fully autonomous robots, e.g., in terms of design,manufacturing and maintenance cost, ease of programming, flexibility andadaptability, and ease of use. Haptic-feedback provided to the humanoperator through the haptic-enabled input device when the human operatorinteracts with the virtualized environment directly (e.g, using avirtual pointer) or through a virtual object allows the user to moreaccurately assess the physical scene of the robot, and more accuratelyprovides guidance to the robot by changing, marking, or otherwiseaugmenting its environment inputs.

Individual features or combinations of features of the haptic-enabledmixed initiative remote robotic control system as described herein invarious embodiments will further the advantages set forth above and/orprovide additional advantages which will be elaborated on in more detailor will be apparent to a person skilled in the art in light the presentdisclosure.

Reference will now be made in detail to implementations, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the various describedimplementations. However, it will be apparent to one of ordinary skillin the art that the various described implementations may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, circuits, and networks have not been describedin detail so as not to unnecessarily obscure aspects of theimplementations.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first userinterface could be termed a second user interface, and, similarly, asecond user interface could be termed a first user interface, withoutdeparting from the scope of the various described implementations. Thefirst user interface and the second user interface are both userinterfaces, but they are not the same user interface.

The terminology used in the description of the various describedimplementations herein is for the purpose of describing particularimplementations only and is not intended to be limiting. As used in thedescription of the various described implementations and the appendedclaims, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

It is to be appreciated that “remote control” may refer to control overa wired or wireless network, such as the Internet, or an intranet withinan organization or household network, the exact distance or locationfrom which the inputs from the human user is provided is notspecifically limited, and the scope of the present teachings is not solimited.

It is also to be appreciated that while the terms user, human operator,administrator, technician, person, and the like may be used to refer tothe person or persons acting in the context of some particularlysituations described herein, these references do not limit the scope ofthe present teachings with respect to the person or persons who areperforming such actions.

FIG. 1 is an exemplary mixed-initiative robotic control environment 100in accordance with some implementations.

In the environment 100, a robot 102 is located at a first location(e.g., local site 104). The robot 102 or the local site is equipped witha set of sensors 106 for capturing visual and depth data of the localsite (e.g., Light Detection and Ranging (LIDAR), or RGBD cameras forcapturing RGB colored image data and corresponding depth data of thescene). The visualized area is the region within the field of view ofthe set of sensors. In some embodiments, the sensors 106 are fixed inlocation relative to the entire scene. In some embodiments, the sensors106 are fixed to the robot, and the field of view of the sensors willchange when the robot moves within the physical environment of the localsite. The robot has onboard processing capabilities to perform low levelfunctions such as moving, grabbing, translating and rotating a grabbedobject, and optionally obstacle avoidance during movement. In someembodiments, the robot has onboard processing capabilities to performintermedia level tasks, such as folding, sweeping, wiping, pressing,opening/closing doors, shaking, stirring, tapping, loading/unloading,etc. sorting, etc. The types of tasks that the robot 102 is designed andpreprogrammed to perform independently (without human intervention) aretailored to the particular application scenarios. For example, adomestic robot will be programmed with capabilities to sweep and mop atarget area when the target area is clearly identified to the robot, andto pick up trash and send them to trash can when the trash items areclearly identified to the robot and the route to the trash is clear andunblocked by other unidentified objects. The robot 102 and the set ofsensors 106 are further equipped with network communication capabilitieswith sufficient bandwidth to transmit the RBGD data collected at thelocal site, and to receive control instructions from a remotely locatedhuman operator 108 and/or a server 110.

At the local site, various physical objects, physical boundaries,physical surfaces, physical paths, and physical passage ways exist atdifferent locations. The different physical objects, physical surfaces,boundaries, paths, passage ways, may also change in appearance, shape,location, and state. For example, a curtain may be open or closed, orsway in a gust of wind; a door may be opened or closed, a liquid spillmay gradually spread on the floor, a patient may change his/her posture,a pet may move around, etc. The physical characteristics of the objects,boundaries, surfaces, paths, and passages include the shape, size,surface texture, weight, friction, flexibility, elasticity, plasticity,response to touch or impact (e.g., breaking, moving, deforming with orwithout the ability for restoration, or being immobile), etc. Theobjects may be identified as targets (e.g., a recipient or receptacle ofan article that is being delivered, an item that is to be picked upand/or moved, an area that is to be cleaned, etc.), obstacles (e.g.,furniture, pet, walls, etc.), things to steer around (e.g., puddles,toys on the floor, wires, sensitive equipment, expensive home decors,etc.), or things that can be run over, pushed away, open or through(e.g., curtains, doors, balls, carpet, etc.) during normal navigationmovement of the robot, and/or things that can only withstand certainactions but other actions (e.g., carpeted area can be swept andvacuumed, but not mopped), etc.

At the remote site 105, a display generation component 112, such as adisplay, a projector, a heads up display or the like, is used to displaya virtualized version of the physical environment captured by the set ofsensors 106. The image and depth data that are streamed from the set ofsensors 106 are used to generate the virtualized version of the physicalenvironment in the form of a point cloud or other three-dimensionalrepresentation of the physical environment (e.g., models includingrepresentations of virtual surfaces and objects), such that thevirtualized version of the physical environment that are visuallypresented at the remote site 105 via the display generation component112 corresponds closely to the state of the physical environment at thelocal site at substantially the same time.

In addition to the display generation component 112, one or more inputdevices (e.g., a touch-sensitive surface, such as a touch-sensitiveremote control, or a touch-screen display that also serves as thedisplay generation component, a mouse, a joystick, a wand controller,and/or cameras tracking the position of one or more features of the usersuch as the user's hands) is utilized by the human operator 108 toprovide inputs and instructions that will be utilized in controlling therobot 102. The one or more input devices include a haptic-enabled inputdevice 114 (e.g., a three-dimensional haptic-enabled pointing device, ahaptic-enabled glove, etc.) that generates force, motion, and/or texturefeedback to the hand(s) of the human operator in accordance withsimulated physical characteristics and physical interactions that occursat a location in the virtualized environment that corresponds to thecurrent movement and position inputs provided via the input device. Forexample, when the movement and position inputs provided via the inputdevice 114 corresponds to movement along a simulated surface in thevirtualized environment corresponding to the floor at the local site104, the haptic feedback generated on the haptic-enabled input device114 will elicit haptic sensations in the user's hands that correspond tothe friction, texture, and hardness of the physical floor. When themovement and position inputs provided via the input device 114corresponds to movement to the edge of the floor and up an abuttingwall, the force feedback provided on the input device will inform theuser of the physical structural relationship between the floor and thewall. When the movement and position inputs provided via the inputdevice 114 corresponds to movement on a gravel path, the haptic feedbackgenerated on the input device 114 will allow the user to get a sense ofwhether the robot will be able to navigate on the gravel path or toppleover due to its unevenness. The real-time haptic feedback in conjunctionwith the visual rendering of the virtualized environment of the localsite allows the human operator to accurately experience and assess thescene, and to provide more accurate and prudent guidance to the robot'snavigation and operation. The virtual representation of the physicalworld combined with the haptic feedback provided during interaction withthe scene give rise to a sense of “mixed-reality” because the scene isneither purely “virtual” in the conventional “virtual reality” sense(e.g., as it closely reflects the physical characteristics of thephysical environment (e.g., in terms of geometry and materialproperties)) nor “reality” in the conventional “augmented reality” sense(e.g., as it is not a visual image of the physical environment, butrather a computed representation that combines RGB image data andcorresponding depth data).

In some embodiments, the input devices at the remote site optionallysupports teleoperation as well, and the human operation can temporarilyswitch to the teleoperation mode at any time for particular difficult orcomplex tasks. At other times, the human operator allows the robot tooperate fully autonomously by not providing any guidance inputs oralteration to the virtualized environment inputs processed by therobot's program logic.

In some embodiments, the human operator 108 uses the haptic-enabledinput device 114 to interact with virtualized surfaces and/or objects inthe virtualized environment, and receive corresponding haptic feedback.Based on the haptic feedback, the user can select a location in thevirtualized environment as the starting location for objectsegmentation. The segmentation with the aid of the user input is moreaccurate and less error prone. In some embodiments, some visual featuresin the virtualized environment are artifacts of lighting and shadows,and/or decorative patterns on surfaces. These are easily distracting toa purely computer-based object segmentation method with no real-timehuman assistance, leading to inaccurate results. With the aid ofreal-time human guidance, the three-dimensional object segmentation isless likely to be side-tracked by these visual distractions, as objectsegmentation is anchored around the point of interest identified by thehuman operator and assisted with corresponding depth data.

In some embodiments, the human operator 108 uses the haptic-enabledinput device to insert one or more virtual objects (e.g., virtual object116) into the virtualized environment. The human operation may interactwith the virtualized environment through manipulation of the virtualobjects in the virtualized environment, such as moving a virtual objectalong a surface and/or pushing a virtual object against another surfaceor object in the virtualized environment. A virtual object may be givenspecific shape, size, and surface characteristics that result indifferent simulated haptic feedback to the human operator on thehaptic-enabled input device 114, when the virtual object is moved alongor pushed against surfaces or objects (virtualized physical objects andvirtual objects) in the virtualized environment. The haptic feedback isgenerated based on physics simulation in accordance with the physicalcharacteristics of the virtualized environment at locations of theinteractions.

In some embodiments, the virtual objects can take on different types ofroles that change the virtualized environment in different ways toinfluence the navigation and operation decisions of the robot indifferent ways. In some embodiments, a first type of virtual objects canserve as a protective shield over a virtualized physical object that iseither stationary or mobile in the scene. For example, a protective domeis placed at the location of a small child or pet and moves with thechild or pet in the virtualized environment. In some embodiments, asecond type of virtual objects can serve as a passage indicator thatoverrides the presence of a physical object that is normally treated asan obstacle or barrier by the robot. For example, the robot will treatthe drapery blocking a door way as an obstacle, and placing a virtualobject that is a passage at the location of the drapery allows the robotto ignore the presence of the drapery and push through the drapery innavigation. In some embodiments, a third type of virtual objects canserve as a measuring aid for measuring the virtualized physical objectsin the virtualized environment and/or provide more accurate routeguidance. For example, the user can place down a first anchor point of avirtual object at a first corner where the floor meets two adjacentwalls (e.g., the user will know where the corner is by touching thescene via the haptic enabled input device 114 and receiving the hapticfeedback), and then extend the virtual object along the edge between thefloor and one of the walls until reaching the other corner. The size ofthe virtual object will inform the user of the dimension of the roomalong the wall. In another example, the user needs to determine whetherthere is enough room between two physical objects in the scene for therobot to pass through with enough safety clearance. As it is difficultto accurately make this determination visually, the user can place avirtual object that has dimensions of the robot plus the addedclearance, and the user can try dragging the virtual object through thegap between the two physical objects in the virtualized environment, thephysics simulation of this interaction will produce haptic feedback tothe user via the haptic-enabled input device that the virtual object isor is not blocked by the two objects in the scene. The user candetermine an alternative route or plan other tasks (e.g., request therobot to moving one of the objects first) based on the haptic feedback.In another example, if a one-meter gap is needed between the hot ovenand the robot, the user can place a 1×1 m² virtual cube in the scene (onthe floor near the oven) and then push the virtual cube until it ispushed against the virtualized oven (e.g., the user will feel theresistance of the virtual cube when the virtual cube has come intocontact with the virtualized oven in the scene), and virtual cube willserve as the virtual wall between the oven and the robot when the robotplans its route). In some embodiments, a fourth type of virtual objectsmay be a series of waypoints that the robot should reach before reachinga selected target (e.g., a designated virtual object or virtualizedphysical object in the scene). As the user moves a pointer object 118along one or more virtualized surfaces in the scene and experiencingsimulated haptic feedback via the haptic-enabled input device 114, theuser can choose to mark one or more points along the movement path, asguidance for the robot 102 to follow. The route guidance provided withthe haptic-feedback generated in accordance with actual physicalcharacteristics (e.g., structure, size, texture, etc.) of theenvironment is more accurate and less difficult for the robot to executein most cases. For example, the user will follow the path of leastresistance when dragging his/her finger or hand toward a target objectin the scene, and the path will be followed easily by the robot. Inother types of interfaces where such haptic feedback is not available,the user may have to draw lines based on visual information alone, anddoes not take into account the surface texture, slope, etc. of the path.

In addition to the equipment collocated with the robot 102 and theequipment collocated with the human operator 108, the haptic-enabledmixed initiative robotic control system includes a central computingdevice 120 (e.g., the server 110) that handles the extensive computationtasks related to visual and haptic data processing and rendering, andgenerating intermediate instructions and workflows that bridge the gapbetween the high-level instructions and environment inputs received fromthe human operator 108 and the low-level instructions executable by therobot 102. The central computing device 120 is connected to the localsite equipment and/or the remote site equipment via one or morenetworks. In some embodiments, the central computing device 120 iscollocated with the robot 102 (e.g., in an industrial manufacturingapplication setting, the human operator is remotely located from therobot and the factory server). In some embodiments, the centralcomputing device 120 is collocated with the human operator 108 (e.g., ina field hospital application setting, the robot is remotely located fromthe doctor and the hospital server). In some embodiments, the centralcomputing device 120 is not collocated with either the robot 102 or thehuman operator 108 (e.g., in a domestic assistance setting, the serveris remotely located from the robot and the human operator).

In some embodiments, the central computing device 120 handles thecomputation related to real-time, simultaneous localization and mapping(SLAM) using real-time dense surface mapping and tracking techniques,such as KinectFusion. In some embodiments, other real-timethree-dimensional modeling methods are used to generate a virtualizedthree-dimensional or pseudo-three-dimensional representation of thephysical environment based on the streaming image and depth datacollected from the local site 104. The central computing device 120generates a point cloud during every time step of the streaming RGBDdata using KinectFusion for visual rendering. In some embodiments,real-time, user-directed object segmentation is incorporated into theKinectFusion data processing pipeline, as needed. The central computingdevice 120 also performs haptic render and physics simulation forinteractions between the user 108 and the virtualized environment, e.g.,via a virtual pointer or via manipulation of a virtual object inside thevirtualized environment. In some embodiments, the virtualizedenvironment is represented by a three-dimensional point cloud. In someembodiments, the virtualized environment is represented by athree-dimensional mesh that includes simulated surfaces that correspondto physical surfaces detected in the physical environment.

In some embodiments, the central computing device 120 includes a pathplanner that generates a path based on the user's marking or virtualobjects inserted by the user. The path planner takes into account boththe characteristics of the virtualized physical environment and theuser's modification and/or augmentation to the virtualized physicalenvironment, and generates the path in accordance with preprogrammedpath planning instructions. In some embodiments, the path planner isimplemented on the robot. In some embodiments, the path plannergenerates predicted positions and commands for the robot to follow.

FIG. 2 is a block diagram illustrating an exemplary architecture 200that includes the mixed-initiative robotic control environment inaccordance with some implementations.

As described herein, reconstruction of a representation of thethree-dimensional physical space in which the robot is operating needsto be performed in real-time with fair degree of accuracy in order forthe user to provide appropriate and timely intervention. In order tofacilitate real-time three-dimensional exploration, segmentation,marking, and interactions with the environment, haptic-rendering andphysics simulation need to be performed in real-time in conjunction withthe visualization of the environment. Since the visual and depth datathat needs to be transmitted and processed is voluminous in nature,network latency needs to be managed in order to provide smooth,real-time operation and control based on such data communicated overnetworks with various latencies.

To address the above technical requirements, a real-time efficient androbust mixed reality platform for mixed-initiative control is proposedto enable haptic interactions with streaming data. A Truncated SignedDistance Function (TSDF)-based haptic rendering method with streamingsurfaces is proposed to ensure the smooth and robust haptic interactionwith virtualized static scenes. An interactive object segmentationmethod is proposed to segment objects quickly and accurately. Differenttypes of haptic interactions are introduced in the mixed realityplatform, and a robot state prediction method is proposed to compensatenetwork delays.

In some embodiments, Fusion-based methods (e.g., KinectFusion) are usedto perform localization and mapping of streaming visual and depth data.According to KinctFusion, the streaming RGBD data from thethree-dimensional camera are fused and saves as a Truncated SignedDistance Function (TSDF). KinectFusion can provide the full-scene densegeometry to enable mixed reality. The present disclosure describesvisual and haptic rendering with streaming surfaces generated byKinectFusion using an improved haptic rendering pipeline includingcollision detection, proxy update and force computation. This improvedmethod is computationally efficient and integrate well with theKinectFusion framework. The improved method also works well forintersecting boundary of two or more planes, ameliorating theinstability of previously disclosed methods for such calculations. Thisis especially significant when working with real-world scenes whichinclude complex geometry with many intersecting boundaries. Whenperforming object segmentation, the improved method allows userinteraction with the reconstructed object surface in the scene, andperforms segmentation in real-time, rather than through offline semanticclassifications. In the present disclosure, haptic interaction is usedto select a target object, an interactive region growing method is usedfor object segmentation using both a color image and a depth image, anda resulting labeled image is fused into the TSDF data structure.

As shown in the FIG. 2, an exemplary mixed reality system architecture200 comprises three layers: a robot layer 202, a user layer 204, and anexecution layer 206.

In some embodiments, the robot layer 202 corresponds to a mobile robot(e.g., robot 102) and a set of RGBD 3D sensors (e.g., 3D camera 106)placed on the top of the robot. This layer is responsible for collectingthe color and depth images in real-time, and sending the data to theexecution layer 206 for scene reconstruction, and visual and hapticrendering. A low-level task executor in this layer execute the controlcommands that are sent by the controller in the execution layer 206 toperform navigation and object manipulation tasks.

In some embodiments, the execution layer 206 receives the RGBD images(e.g., streamed from the RGBD sensors on top of the robot 102 over anetwork) and performs simultaneous localization and mapping (SLAM) usinga KinectFusion module. The KinectFusion module generates a point cloudevery time step for visual rendering on a display monitor. The executionlayer 206 combines an object segmentation module with the KinectFusionmodule to segment and mark an object in the scene as required by theuser. The execution layer 206 also includes a separate thread for hapticrendering that is performed in parallel with the visual rendering (e.g.,generation of the point cloud). A physics simulation module handles thesituation that the haptic interaction interface is used to interact witha virtual object in the scene and/or to interact with the scene throughmanipulation of a virtual object in the scene. The haptic rendering andphysics simulation produces haptic feedback data that controls thehaptic-enabled input device (e.g., input device 114) held by the user(e.g., user 108). The haptic feedback includes position, motion, force(e.g., reactive force such as resistance to pressure and impact, andfrictions), texture, vibrations, etc., output to the user's hand via thehaptic-enabled input device (e.g., input device 114). In someembodiments, the execution layer 206 includes a path planner thatgenerates a path based on the user's marking inputs and/or the virtualobjects placed within the scene, in addition to the characteristics ofthe virtualized environment deduced from the image and depth data (e.g.,as represented by the TSDF or point cloud). In some embodiments, theexecution layer 206 includes a controller module that is used togenerate a predicted position and commands for the robot to follow. Insome embodiments, the path planner and the controller are optionallyimplemented in the robot layer 202 and receives the user input data andthe environment data from the execution layer 206. In some embodiments,the execution layer 206 is implemented on the central computing system120 (e.g., a server 110).

In some embodiments, the user layer 204 provides all the interactioninterfaces and outputs to the user (e.g., user 108). The user can eitheruse a teleoperation interface to directly operate the robot (e.g.,directly providing low level commands and instructions), or use thehaptic interfaces to interact with the virtualized three-dimensionalenvironment. In some embodiments, a haptic guided object segmentationinterface is provided to the user and is only used for segmentation. Thehaptic interaction interface enables the user to use haptic input toexperience and realize placement and movement of virtual objects in thescene, and assigning various types of roles to the virtual objects(e.g., target, passage, obstacle, shield, guide, measure, etc.). Hapticmarking is used to either define a path on a surface in the scene ormark one or more waypoints (e.g., virtual object and/or virtualizedphysical object) for the robot to follow, approach, and/or track. Insome embodiments, the marking and movement of the virtual object isprocessed in real-time in the virtualized scene and followed by therobot in real-time in the physical scene. The virtual obstacle interfaceenables the user to add virtual objects of any form of geometries intothe scene, then the path planner will search a new path in accordancewith the positions and the assigned roles of the virtual objects, inconjunction with the characteristics of the scene. In addition to hapticfeedback, the visual rendering also reflects the interactions with thescene by the user, either directly through a virtual pointer orindirectly through interaction with a virtual object in the scene.

FIG. 3 illustrates an exemplary processing pipeline 300 for real-timevisual and haptic rendering at the remote site of the mixed-initiativerobotic control environment, in accordance with some implementations.

As shown in FIG. 3, RGBD data 302 received from the sensors (e.g.,camera 106) collocated with the robot (e.g., robot 102) is streamed overa network and provided to a graphical processing unit 302 (GPU) on acentral computing system 120 (e.g., server 110). Since KinectFusion isapplied for dense mapping and localization, dense geometry is generatedas the streaming surfaces of the 3D virtualized environment. In the GPU302, TSDF update is performed, followed by collision detection, andfollowed by proxy update with force shading and ray casting. Theresulting data includes a point cloud 306 and proxy values 308 for allhaptic interaction points (HIPs). The point cloud 306 and the proxyvalues 308 are utilized in visual rendering on a display (e.g., display112), and the proxy values 308 are utilized in force rendering on thehaptic-enabled input device (e.g., input device 114).

In the present disclosure, a novel proxy update method with forceshading is proposed, which is more efficient and guarantees the stablerendering at intersecting boundaries of different planes. Furthermore,the method allows addition of surface properties such as friction andhaptic textures in haptic rendering.

Proxy update is a key-part of constraint-based haptic rendering, sincethe proxy is not only used to compute the force at an interaction point,but also rendered visually to the viewers. If the proxy update is notstable and smooth, the force rendering and visual rendering will not besmooth. Previously, a proxy update method that has been published usesgradient-based method to find the nearest surface point. As shown inFIG. 4, FIG. 4(1) on the left shows a scenario that the hapticinteraction occurs on a surface with a sharp change in direction (e.g.,interaction goes around a corner of a curved surface or the boundary oftwo intersecting planes). In this scenario, the haptic interaction point(HIP) is moved by the user from h_(i−1) to h_(i), and the proxy positionis changed from p_(i−1) to p_(i), in accordance with previouslydisclosed haptic rendering methods. Since the proxy is always thenearest surface point according to the HIP in the previously disclosedmethod, the proxy undergoes a sudden change in position. In terms ofuser experience, it would feel as though the user's finger has suddenly“jumps” to the other side of the surface, and computed force is changeddrastically to an almost reversed direction. This feels distracting,unrealistic, and confusing to the user.

Force shading is an algorithm for smooth haptic rendering. In thisdisclosure, a novel TSDF-based proxy update method with force shading isproposed. Different from previous force shading methods, the presentlydisclosed method focus on the TSDF data structure, which can be used inall of the fusion-based 3D reconstructions. Two scenarios are handled inthe improved proxy updating method:

a. If the HIP is the first contact with the surface in an interaction,the proxy is to find the nearest surface point. Instead of thegradient-based iterative method proposed in previously disclosedmethods, the task of finding the nearest surface point is integratedinto the ray casting step in KinectFusion. The reason for this change isthat the deformable property for the surface is not considered, andtherefore the ray casting is performed after the haptic rendering.Per-pixel ray marches in TSDF to generate the point cloud for the wholesurface. During this procedure, the distances between the HIP and everypoint on the surface are computed and saved. The nearest surface pointfinding problem now becomes a parallel problem that finds the minimum inthe distance array. This problem can be solved through parallelreduction as disclosed in “SC07, High Performance Computing with CUDA(2007) by Mark Harris.” In some embodiments, nearest surface pointfinding in ray casting require the following: Given a starting pointh: 1. Parallelized thread: each pixel's corresponding ray; 2. Marchesfrom minimum depth, stop when zero crossing to get surface points; 3.Compute the distance d=|s-h|; 4. Parallel reduction to get minimumdistance, and return the corresponding surface point as the nearestpoint.

b. After the HIP penetrates into the surface, the subsequent proxyposition needs to be updated since the HIP will penetrate further intothe volume. As shown in FIG. 4, the nearest surface point is notappropriate for this scenario, a more correct way is to constrain thesucceeding proxy. In the presently disclosed method, the previous timestep normal n_(i−1) is used to define a tangent plane (as shown in FIG.4(2)), the normal of the proxy will be computed every time step.Tracking this normal is like tracking a tangent gliding plane over thesurface physically. As shown in FIG. 4(2), the tangent plane T_(i−1) is“dragged” by the new proxy position h_(i) while attached on the surface.So, the tangent plane can be treated as a constraint plane for theproxy. First, a perpendicular is dropped from h₁ to this constraintplane to get a goal position g_(i) , which is the first approximation ofthe proxy. Then, the nearest surface finding in the ray casting step(see description in section a above) will be used to find the new proxyp_(i). The core of this method is to use the tangent plane to constraintthe new proxy in a physically plausible way, then to refine it as thenearest surface point. In some embodiments, the whole procedure forsubsequent proxy update with force shading includes: 1. Based on thenormal n_(i−1), get the tangent plane T_(i−1); 2. Drop a perpendicularfrom the current HIP h_(i) to T_(i−1) to get g_(i), and 3. Use thenearest surface point finding in ray casting algorithm described above,with g_(i) being used as the first point, to compute the final proxyp_(i).

In some embodiments, the improved method described herein simulatesurface properties to generate simulated friction forces, and differenthaptic sensations of textured surfaces. In some embodiments, thefriction force can be simulated by a simple change using a knownfriction cone. The angle a defines a cone starting from the current HIPh_(i), as shown in FIG. 5. The friction cone forms an interaction circlewith the tangent plane from the previous proxy point p_(i−1). In someembodiments, β=arctan(μ), where μ is a user defined frictioncoefficient. If the previous time step proxy p_(i−1) is inside theinteraction circle (e.g., as shown in FIG. 5(1)), then the new proxywill be directly set to the same value as before: p_(i)=p_(i−1). If theprevious time step proxy p_(i−1) is outside of the interaction circle(e.g., as shown in FIG. 5(2)), then the goal position (approximatedproxy) g_(i)=c_(i), where c_(i) is the point closest to p_(i−1) on theinteraction circle. These two scenarios correspond to simulated staticfriction and simulated dynamic friction. In some embodiments, the haptictexture is implemented by using a bump texture method which can generatethe constraint for each point on the surface to change the normal.

In some embodiments, an interaction region growing 2D segmentationmethod is used to perform object segmentation with haptic-enabled userguidance. To segment a three-dimensional object from the reconstructedthree-dimensional scene, a haptic-enabled segmentation interface isprovided. Such an interface enables more flexible haptic interaction,e.g. haptic texture, material properties for different objects, and alsoprovides the object position and orientation for robot grasping tasks.Many previous researches combine object detection and semanticclassification into dense mapping. The presently disclosed system aimsto build haptic-enabled interfaces for the mixed-initiative control,therefore the high-level semantic segmentation is not required. Thepresently disclosed interactive 3D object segmentation method is notonly efficient, but also compatible with the popular high-level objectsemantic algorithms as the input.

In some cases, the straightforward way to perform object segmentation isto segment the 3D object from the 3D point cloud. It is also possible touse the KD-tree to speed up the neighbor search for points. However,these conventional methods take extra processing time. An alternativeway is to perform the segmentation based on TSDF data, and save thesegmentation information into the TSDF.

In the presently proposed KinectFusion pipeline (as shown in FIG. 3),the depth image is fused for surface reconstruction at each time step.Thus, a two-phase algorithm as follows may be proposed, as shown in FIG.6. In the first phase, 2D segmentation is performed from both the depthimage and color image. After the 2D segmentation, a label image L_(i) isgenerated. In the second phase, the segmentation is fused into the TSDFtogether with the depth image (e.g., in the TSDF update step in FIG. 3).In this way, the segmentation is seamlessly integrated into theKinectFusion processing and reduces the overall computation time cost.Moreover, the segmentation information will be fused by weight, whichgenerates a robust segmentation result.

In some embodiments, during the first phase of the improved method, theuser uses the haptic avatar (e.g., a virtual pointer) to “touch” andmark an object of interest in the 3D scene. Then the 3D mark (e.g., withits 3D coordinates) is transformed into coordinates in the current colorimage. In the next time step, starting from the marked point in theimage, the pixels are clustered through a region growing method untilthere are no pixels to be added to the region of the marked point. Theregion is treated as a cluster, then the distances between a neighborpixel and the cluster center is computed as the combination of twoEuclidean distances as shown Equation 1 below:d(x _(i) , S)=∥I(x _(i))−I(S)∥₂ +β∥P(x _(i))−P(S)∥₂   (1)

where x_(i) is the neighbor pixel's position, and S is the center of theregion. I is the CIELAB color space value of the pixel in the colorimage, which is widely considered as perceptually uniform for smallcolor distances. P is the 3D coordinates which are computed from thedepth image. The values for cluster center: I(S) and P(S) are computedas the averages of the values of all pixels in this cluster. β=m/g is aparameter that controls the compactness of a region. m is the variableto control the compactness, g is the grid interval. The region growingwith RGBD data provides better boundary of objects that only with RGBdata. The greater the value of m, the more spatial proximity isemphasized and the more compact the cluster. This value can be in therange [1, 20], e.g., a value of 10 is commonly used. The distancethreshold can be chosen by the user based on characteristics of thescene and/or the object of interest in the scene at the time ofsegmentation.

FIGS. 7A-7C illustrate various examples of using haptic markings andvirtual objects to influence route planning, in accordance with someembodiments.

In most previous works, haptic force feedback is used to generate pathguidance forces, collision preventing forces, or environmental force toimprove the performance of high-level tasks operations. However, thepresently disclosed system uses a haptic device in a different way. Ahaptic device is used as the 3D avatar to remotely touch, explore andinteract with the virtualized real-world environment. The hapticinteraction provides more flexible operations similar to using “virtualhands”. The haptic interfaces can intervene the robot control procedure,and add a new path or change destinations. These interfaces will notinfluence the velocity of the robot, but only the paths and targetpoints.

In some embodiments, a haptic marking interface (e.g., user interface700) is provided for providing path guidance to the robot, since hapticrendering for surfaces in the scene is performed in real-time and iscomputationally efficient. Through the haptic marking interface, a usercan use an HIP to touch a surface (e.g., a floor) in the scene to mark apath for the robot to follow. Then, the control manager in the executionlayer takes this marked path as input to invoke the path planning. Insome embodiments, the marking is saved as an ordering point set, andsaved separately in the remote server. In some embodiments, the guidanceis provided as an option to the path planner, for the path planner toconsider in light of other options available in the scene. In someembodiments, when multiple options are available to the path planner,the user's marking is given more weight or dominant weight relative toother options available to the path planner. In this manner, thepreprogrammed path planning instructions can be used normally, with orwithout user intervention, and the robot control can slide between fullyautonomous to teleoperation, or anywhere in between, at any time,thereby improving the flexibility and effectiveness of the roboticcontrol.

In some embodiments, the presently disclosed system provides a hapticmarking interface (e.g., user interface 700) that further allows theuser to add one or more virtual objects to the 3D virtualized scene toindirectly influence path and/or operations of the robot, in accordancewith some embodiments. It is much easier to use the haptic-enableddevice to place a virtual object at a desired location in the scenebecause of the haptic feedback that conveys the absolute physicalcharacteristics of the scene at different locations (e.g., dimensions,texture, slope, etc.), and the characteristics relative to a virtualobject or the robot. This enables the user to locate a suitable 3Dposition for the virtual object quickly and accurately, or to guide theuser's the placement of the markings that are used to provide pathguidance to the robot.

As shown in FIG. 7A, the example haptic marking interface 700 includes avisual rendering region 702 that displays the virtual representation ofthe physical environment in front of the robot. The virtualized physicalenvironment represented in the visual rendering region 702 is optionallya point cloud that is dynamically updated in real-time with thestreaming image and depth data received from the sensors collocated withthe robot. In some embodiments, the points displayed in the visualrendering region 702 have been segmented (e.g., using the user-assistedregion growth method disclosed herein), and surfaces and boundariesassociated with physical objects in the point cloud have been labeledand visually represented as such. For example, the floor is segmentedfrom a user-specified point 805, after the user has explored the floorregion in the image using the haptic enabled input device.

As shown in FIG. 7A, a cubic box has been segmented out of the pointcloud, and an outline 704 of the box is shown around the box. The userhas marked the box as a target for the robot's next task, by selectingthe outline 704 and selecting an object type of “target” for theoutline. For example, by invoking a contextual menu while the box isselected, and selecting the “target” object type from the contextualmenu. When moving the virtual pointer 705 on the surface of the boxusing a haptic-enabled input device, the haptic feedback provided on thehaptic-enabled input devices convey the geometry of the box (e.g.,turning around the intersection of abutting planes of the box).

As shown in FIG. 7A, the user has also placed a series of waypoints 708on the surface of the floor shown in the visual rendering region 702.For example, the user can select the object type “waypoint” from theobject type selection region 706 (e.g., a drop-down menu of variousvirtual object types, including waypoint, target, obstacle, shield,passage, measure, etc.). Then, the user can move the virtual pointeralong the floor surface and selectively putting down a series ofwaypoints along a movement path of the virtual pointer. The hapticfeedback provided on the haptic-enabled input devices convey thefriction and surface geometry of the floor along the movement path ofthe virtual pointer, and also constrain the movement path of pointer(e.g., the pointer cannot go through the floor, and the smoothness andspeed of the pointer movement is constrained by the roughness of thefloor).

In an example, the proposed system provides a haptic-enabled interface(700) from which the user can select or specify one or more virtualobjects (e.g., virtual objects with preset dimensions or shapes, such asspheres, domes, cubes, cones, lines, walls, tables, chairs, etc., ofdifferent sizes) that user can drag into the 3D rendering of thevirtualized scene (e.g., visual rendering region 702). In someembodiments, a virtual object with the shape and size and/or otherphysical characteristics of the robot is also available in the interfacefor the user to drag into the 3D scene. In some embodiments, the usercan select the virtual object and resize it within the scene by settingan anchor point on one side of the virtual object and drag anotheranchor point on another side of the virtual object within the 3D sceneto resize it.

As shown in FIG. 7A, the user can select an object type of a virtualobject from the object type selection region 706. The user can alsoselect an object shape from the object shape selection region 710. Theobject shape selection region displays a listing of various shapes, andan option for the user to draw a customized shape. In some embodiments,the object shape selection region includes affordances (e.g., checkboxes) for specifying whether the specified shape is a solid shape or ahollow shell.

As shown in FIG. 7A, a user can also specify the surface material andsurface texture of the virtual object in surface material selectionregion 712 and surface texture selection region 714. The surfacematerial and surface texture information is used when generating theforce feedback provided to the haptic enabled input device, when theuser interacts with the virtual object or when the user interacts withthe environment through manipulation of the virtual object. In someembodiments, the user can also specify a weight (e.g., a mass) for thevirtual object that affects the haptic rendering involving the virtualobject. In some embodiments, a virtual object may be a composite objectwith different portions corresponding to different shapes, materials,textures, and/or weights, respectively.

In some embodiments, as shown in FIG. 7A, the user interface 700 alsoincludes a staging region 716. The user can drag a default shape intothe staging region 716 and adjust the dimensions and orientation of theshape. The user can also compose a composite object by dragging multipleobject shapes into the staging region and adjust the relative size,location, and orientation of the shapes. Once the user is satisfied withthe size, shape, and orientation of the object in the staging region716, the user can drag and drop it into the visual rendering region 702(e.g., onto the floor in the scene).

In some embodiments, after the user has selected a virtual object toplace into the 3D scene, the user may explore the 3D scene via thehaptic-enabled input device before dropping the virtual object into thescene at a specific location in the scene. For example, after the userhas selected to drop a dome into the scene, an outline of the dome isshown in the 3D scene around the user's 3D avatar (e.g., virtualpointer) in the scene. The user can drag the 3D avatar around varioussurfaces in the scene, to experience the slope, texture, dimensions ofthe scene via the haptic-enabled input device. When a user hasidentified a suitable location, the user can provide an input to dropthe virtual dome at that location. As shown in FIG. 7B, a virtual dome718 has been dropped into the virtualized environment shown in visualrendering region 702 of the user interface 700.

In some embodiments, if the user has moved into a region that cannotaccommodate the virtual dome (e.g., the space is too narrow for thedimension of the virtual dome), or if the location is already occupiedby a physical object that is too large to be placed under the virtualdome, a warning feedback (e.g., visual alert) is provided to the uservisually on the visual rendering of the 3D scene, and/or through hapticson haptic-enabled input device. In some embodiments, different types ofvirtual objects are associated with different types of restrictions onwhat types of space or physical characteristics of space are compatiblewith the placement of the virtual object. For example, a solid objectcan only be placed within an empty space that is larger than the solidobject's dimensions; while a shell object can be placed within an emptyspace that is larger than the shell object's dimensions, or a space thatis occupied by a physical object that is smaller than the innerdimensions of the shell object. Also, the orientation of the virtualobject is adjustable by the user before it is placed into the scene. Forexample, when a user tries to drag a virtual couch through a doorway inthe 3D scene, the virtual couch will be blocked until the user rotatesthe virtual couch to its side. This is useful when the user wants todetermine the robot can actually make a real couch of the samedimensions go through the doorway if provided with correct instructions(e.g., rotation first before pushing).

In some embodiments, physics simulation may be performed in addition tohaptic rendering at the location of the virtual object. The user maydrag a virtual robot up an incline in the 3D scene and experience theslope of the surface, and the surface friction, and optionally, somerepresentation of the weight of the robot (e.g., visual indication ofthe robot's power usage for driving up the incline). When the virtualrobot is set down on the surface, the physics simulation may show therobot staying in place, or slide down the incline, or topple overdepending on how steep the incline is.

In some embodiments, when the virtual object is placed on a simulatedphysical surface in the 3D scene that corresponds to a physical surfacein the real-world, the ground plane is located and saved at the firstseveral time steps. The virtual object can be assigned one or moreroles, such as obstacle, shield, wall, target, passage, safety cushion,etc. The path planner will regenerate the new path to accommodate thechanges to the virtualized scene with the addition of the virtualobject. For example, an insertion of an obstacle type virtual object maycause the path planner to bend an original path for the robot to goaround the obstacle. An insertion of a shield type virtual object thatis placed over a physical object moves with the physical object in the3D scene, and the path planner ensures that the robot is always outsideof the shield. An insertion of a wall means that the path planner willallow the robot to be on either side of the wall, but cannot move acrossthe wall. An insertion of a target type virtual object causes the pathplanner to modify a path to approach the target virtual object. In someembodiments, when multiple virtual objects are designated as targetobjects, the user optionally assign an order of the target objects viathe haptic marking interface such that the path planner approaches thetarget objects in the specified order. In some embodiments, when theorder is not explicitly specified otherwise, the path planner plans thepath in accordance with predefined criteria set in the preprogrammedpath planning instructions (e.g., based on the order that the virtualobjects are inserted into the scene, based on the shortest path to reachall target objects, based on increasing/decreasing sizes of the virtualobjects, etc.). In some embodiments, a target type virtual object isplaced on a physical object or a portion of a physical object in the 3Dscene, such that when the robot is instructed to manipulate the virtualobject (e.g., grab and lift, or grab and rotate), the physical object orthe portion of the physical object at the location of the virtual objectis manipulated accordingly by the robot. In a more specific example, atarget virtual object is placed on an edge of a revolving door, and whenthe robot approaches the edge of the revolving door and pushes on thetarget virtual object in accordance with instructions received from thepath planner and controller, the revolving door will swing open. If nosuch virtual object is placed on the edge of the revolving door or whenthe virtual object is placed in the middle of the revolving door, therobot will not approach the revolving door (e.g., for seeing it as anobstacle) or will crash into the revolving door (e.g., for failing toopening it by pushing in the middle of the revolving door as opposed tothe edge of the revolving door). In the above example, the preprogrammedalgorithms and logic for generating the path plan and objectmanipulation instructions remain unchanged, and the resultinginstructions sent to the robot are different because of the differencein environmental information received by the robot (e.g., the differenceintroduced by the virtual object(s)' presence/location(s)). In someembodiments, an insertion of a passage type of virtual object at alocation of a virtualized physical object in the 3D scene causes thepath planner to disregard the physical object as an obstacle in pathplanning. For example, the path planner will avoid going through adoorway covered by drapery in accordance with its preprogrammed obstacleavoidance instructions; however, when a passage type virtual object isplaced on the drapery via the haptic marking interface by a user, thepath planner will no longer treat the drapery as an obstacle whenplanning a path toward a target object on the other side of the doorway. If the drapery happens to partially block the robot, the robot willpush through the drapery to go through the doorway (e.g., the closed orhalf open drapery will be pushed open to let through the robot) whenexecuting the path. In some embodiments, an insertion of a passage typevirtual object that is placed over a physical object moves with thephysical object in the 3D scene, such that the path planner does nottreat the physical object as an obstacle irrespective of the change inlocation of the physical object. For example, the path planner willavoid a small rubber ball lying on the floor in accordance with itspreprogrammed obstacle avoidance instructions; however, when a passagetype virtual object is placed and attached on the virtualized rubberball via the haptic marking interface by a user, the path planner willno longer treat the rubber ball as an obstacle when planning a pathtoward a target object in the scene. If the rubber ball happens to be inthe path, the robot will approach and push against the ball (which willroll away or be pushed along) when executing the path.

In some embodiments, the haptic-marking interface allows the user tomanipulate a virtual object already placed in the virtualized 3D scene,and provide haptic feedback to the user via the haptic-enabled inputdevice that mimics reactions from the 3D environment through the virtualobject.

In some embodiments, after a virtual object is placed on or against asurface in the virtualized physical environment, the user can interactwith the virtual object, and experience different haptic feedbackdepending on the manner by which the user is interacting with thevirtual object, and the spatial relationship between the virtual objectand the virtualized physical environment, as well as physicalcharacteristics of the virtual object and the virtualized physicalenvironment. For example, a virtual object can be placed on a floorsurface in the virtualized physical environment, and pushed or draggedon the floor surface; and the haptic-enabled input device provideshaptic feedback that simulates the static and dynamic frictions betweenthe virtual object and the floor surface. In another example, a virtualobject can be pushed against a surface (e.g., sideways against a wall oranother object, or downward against the floor, etc.), and thehaptic-enabled input device provides haptic feedback that simulates thereactive force from the surface against the virtual object.

In FIG. 7B, a few examples are provided to illustrate the influence topath planning by the placement of various types of virtual objects(e.g., dome 718) into a virtualized physical environment, in accordancewith some embodiments. FIG. 7B(a) illustrates a path 720-1 (e.g., aroute of least resistance and distance) of a robot 102 would take toreach a target object 704. The path 720-1 is generated by thepreprogrammed path-planning instructions of the path planer implementedat the robot or the server in accordance with the environmentinformation provided to the path planner. FIG. 7B(b) illustrates a path720-2 of the robot 102, when a series of waypoints 708 are laid down onthe floor of the virtualized environment. The path 720-2 first goesthrough the series of waypoints and then turns toward the target object704. The environment information provided to the path planner is changedfrom the scenario in FIG. 7B(a), and resulting in the change in theplanned path. FIG. 7B(c) illustrates that a dome shaped virtual object718-1 is placed in the scene, and the dome is of the “obstacle” objecttype. The path 720-3 is generated to avoid the obstacle 718-1 whenmoving toward the target 704. Even though the virtual dome does notexist in the physical environment, the environment information providedto the path planner is modified to include the presence of the dome, andas a result, the path planner generates a path that goes around thedome. FIG. 7B(d) illustrates that a dome-shaped shield 718-2 is placedover a pet that moves about the room. The dome-shaped shield 718-2follows the pet's movement in real-time, and the path planner adjuststhe path in accordance with the location of the dome shield 718-2. Theenvironment information is continuously updated with the location of thepet (e.g., geometric center of the pet), but the calculation is greatlysimplified in real-time, because even though the pet's shape continuesto change (e.g., moving, running, rolling, etc.), the details of thepet's shape is replaced with the dome shield's shape, and the pathplanner will plan a path to steer clear of the dome, as opposed to thepet. FIG. 7B(e) illustrates that, there is a crack 722 on the floorwhich appears to be an obstacle to the path planner because thecontinuity of the floor is disrupted at the crack. However, when apassage dome object 718-3 is placed on a portion of the crack 722, theenvironment information provided to the path planner is modified, andthe gap in the floor surface is bridged, and the path 720-5 is goesthrough the dome passable object over the crack, and goes directlytoward the target 704. FIG. 7B(f) illustrates that, there are someobstacles 724 on the way to the target. The user uses the dome shapedmeasure object 718-4 to push through the gaps between obstacles. Thesize of the measure object 718-4 is selected to be the larger of therobot and the target, if the task is to move the target to a differentlocation. In one location, the measure object 718-4 is stuck (e.g., theuser will not be able to push through), and in another location, themeasure object can pass through without problem. Thus, the user can,after trying out the two locations by pushing the dome measure object718-4 against the two openings, either reset the object type at thenarrow opening to obstacle, or reset the object type at the wide openingto passage, to cause the robot to move to the target through the wideopening. The path 720-6 is generated based on the modified environmentinformation provided to the path planner.

FIG. 7C illustrates movement of a virtual object (e.g., a cube 726) inthe virtualized physical environment, in accordance with someembodiments. As shown in FIG. 7C, the virtual object is pushed along thefloor and ultimately against the box 704 in the virtualized physicalenvironment. As the user pushes the virtual object, the movement of thevirtual object is constrained by the geometry of the floor surface inthe virtualized environment, and when the virtual object is pushedagainst the physical box in the scene, the movement of the virtualobject is stopped. The haptic enabled input device provides forcefeedback that corresponds to the friction between the virtual object andthe floor surface when the virtual object is pushed along the floorsurface, and the reactive force from the physical box 704 when thevirtual object is pushed against the physical box.

In some embodiments, the presently proposed mixed initiative remoterobotic control system includes mechanisms that take into considerationof network latency when performing visual and haptic rendering, pathplanning, and/or physics simulations. In some embodiments, thelatency-based adjustment is automatically triggered by the detection oflatency exceeding a threshold amount (e.g., the network latency ismonitored continuously) or when the user provides a request foradjusting for latency after seeing the effect of network latency (e.g.,unexpected and/or unpredictable robot movement, or dithering movement ofthe robot) or an indicator value of the network latency (e.g., 100 ms,or 200 ms) through other latency monitoring systems.

The reasons that latency is an important consideration in path planningand operation control of robot for mixed-initiative remote roboticcontrol include the following: (1) the amount of image and depth datathat need to be transmitted over the network is very large; and (2) theuser intervention is not continuous, but intermittent and unscheduled,and the robot has much time executing autonomous motion. To accommodatethese characteristics, the control architecture of the system will bedistributed in both the robot layer and the execution layer.

In some embodiments, the execution layer (e.g., execution layer 206)includes a task planner, a path supervisor, a path planner, and aprimitive supervisor. The robot layer (e.g., robot layer 204) includes atrajectory planner, a controller and the robot. A user uses hapticinterfaces to invoke high-level tasks, including haptic marking aposition or insertion of a virtual object in the 3D scene, and hapticinteraction with a virtual object. These operations are passed to thetask planner. The task planner is a high-level manager to communicatewith the plan supervisor. It can parse the task into micro-action plans,and receive the re-planning request. The plan supervisor can request andreceive the path between two points from the path planner. In thepresently disclosed framework, the path generation is based on aRapidly-exploring Random Tree algorithm, in some embodiments. In amixed-initiative control scenario, the human-in-the-loop interactionshappen in the primitive supervisor module. The low level primitivesupervisor receives the path information such as waypoints and microactions from the plan supervisor. It will receive the planned path, andalso the haptic marking path, and generate a goal position for the robotmotion. In the robot layer, the trajectory planner monitors and controlsthe trajectory towards the goal position. The haptic marking pathprovides a marking point x^(m), and the planned path provides a pathpoint x^(p). The goal position x^(g) is chosen from these two points bychoosing the maximal distance between the point and current robotposition.

Network delays may influence the mapping and localization fromKinectFusion. To compensate the delay, a method to generate a predictedgoal position is proposed. Assuming current velocity of robot isv_(i)=(a_(i), b_(i), c_(i)) at i_(th) time step, a way to predict thenext velocity is to compute the velocity and acceleration with the lastseveral frames. Most Kalman filters are based on an empirical model ofthis linear form. A general linear model used to predict next velocityv_(i+1) is Equation 2:a _(i+1) =α ₀ a _(i) +a ₁ a _(i−1) . . . α _(m) a _(t−m) b _(i+1)=β₀ b_(i)+β₁ b−1 . . . β_(m) b _(t−m) c _(i+1)=γ₀ c _(i)+γ₁ c−1 . . . γ_(m) c_(t−m)   (2)

For a given time series of points in a path, the matrix V is defined asEquation 3:

$\begin{matrix}{V = \begin{bmatrix}a_{0} & \ldots & a_{m} & b_{0} & \ldots & b_{m} & c_{0} & \ldots & c_{m} \\a_{1} & \ldots & a_{m + 1} & b_{1} & \ldots & b_{m + 1} & c_{1} & \ldots & c_{m + 1} \\\vdots & \; & \mspace{11mu} & \; & \; & \; & \; & \; & \; \\a_{i} & \ldots & a_{i + m} & b_{i} & \ldots & b_{i + m} & c_{i} & \ldots & c_{i + m}\end{bmatrix}} & (3)\end{matrix}$

Let v be the predicted position (v_(m+1), v_(m+2), . . . , v_(i+m+1), .. . )^(T). The problem now is to solve and obtain three parametervectors α, β, and γ. The general solution of these linear problems areshown as follows in Equation 4:α=(V^(T)V)V^(T)a β=(V^(T)V)V^(T)b γ=(V^(T)V)V^(T)c   (4)

Every time step, this linear prediction model generates new parameters,and then predicts the next goal position x^(g) _(i+1)=x^(g)_(i)+v_(i+1)t, where t is the round time delay. This goal position willbe sent to the trajectory planner for the low-level autonomous controlof the robot.

FIG. 8 is a flowchart of a method 800 of providing mixed-initiativerobotic control, in accordance with some embodiments. The method 800 ofproviding mixed-initiative robotic control is performed (802) at acomputing device having one or more processors and memory. The computingdevice communicably coupled to a robot and is configured to generate aplanned path for the robot in accordance with a first set ofpreprogrammed path-planning instructions. The robot is configured tonavigate within a physical environment in accordance with the plannedpath received from the computing device and locally-storedpath-execution instructions. In some embodiments, the computing deviceis a server that is collocated with the robot (e.g., using the samelocal area network or in the same room). In some embodiments, thecomputing device is a server that is remotely located from the robot,and communicates with the robot over a wide area network, and mayexperience network delays when communicating with the robot over thenetwork. In some embodiments, the computing device is a computer that iscollocated with a human operator that facilitates remote interactionbetween the user and the robot.

In accordance with the method 800, the computing device displays (804) acontrol user interface (e.g., a control interface 700 that includes agraphical rendering 702 of the virtualized scene (e.g., displaying thepoint cloud or surfaces identified based on the image and depth datacollected from the sensors at the location of the robot, e.g., with aframe rate corresponding to substantially real-time update) and varioususer interface controls (e.g., controls in 706, 710, 712, 714, and 716in user interface 700 in FIG. 7A) for marking the virtualized scene andinserting virtual objects into the virtualized scene) via a displaygeneration component (e.g., a display, a heads-up display, a projector,a touch-screen display, etc.) coupled to the computing device. Thecontrol user interface displays a virtual pointer object (e.g., virtualpointer 705) (e.g., a virtual hand, a cursor, an arrow-shaped positionindicator, etc. that moves in the virtualized scene in accordance with amovement input of the user (e.g., provided by a mouse, joystick,touch-pad, haptic-enabled pointing device, etc.)) within a virtualizedenvironment corresponding to a first physical environment currentlysurrounding the robot, wherein the virtualized environment is generatedand updated in accordance with streaming environment data (e.g.,streaming RGB image data and corresponding depth data, with a videoframe rate (e.g., 10, 12, 24 frames per second)) received from a firstset of sensors (e.g., RGBD camera 106) collocated with the robot (e.g.,robot 102) (e.g., the first set of sensors may be affixed to the robot(e.g., on top of the robot, or at the “eye”-level of the robot, or onthe ceiling of the room in which the robot is located)).

While displaying the virtual pointer object in the virtualizedenvironment, the computing device detects (806) first user input via ahaptic-enabled input device. The first user input includes a firstmovement input and one or more marking inputs at one or more locationsalong a movement path of the virtual pointer object in the virtualizedenvironment. For example, a user may operate the haptic-enabled inputdevice (e.g., a haptic-enabled 3D mouse 114) by moving a positioningportion of the input device in 3D, and cause corresponding movement ofthe virtual pointer object (e.g., pointer 705) in the virtualizedenvironment. Based on the location of the virtual pointer object in thevirtualized environment (e.g., as shown in the visual rendering region702 of the control user interface 700) and based the haptic feedbackprovided to the user via the haptic-enabled input device (e.g., thehaptic feedback causes the user to experience force and/or vibrationssimulating reactive force (e.g., from impact, resistance topush/pull/press, etc.), dynamic and/or static friction between user'sfinger and the surfaces in the physical environment, textures ofsurfaces, slope of the surfaces, other physical geometricalcharacteristics of the surfaces, etc.), the user can determine when tomark the location as a waypoint or destination point or other types ofmarking points (e.g., handle for object manipulation, passage for goingthrough a physical obstacle, etc.) and provides the required markinginput (e.g., clicking a button, double tapping on a touch surface,activating a marking control in the control user interface, etc.) viathe haptic-enabled input device (e.g., haptic-enabled input device 114).

In response to detecting the first user input (810): the computingdevice moves (812) the virtual pointer object along a first movementpath in the virtualized environment in accordance with the firstmovement input, where the first movement path is constrained by one ormore simulated surfaces in the virtualized environment. For example, thevirtual pointer object can only move on the surface of the floor, andcannot penetrate the floor when moving along the floor represented inthe virtualized environment.

In response to detecting the first user input (810): the computingdevice generates (814) haptic feedback via the haptic-enabled inputdevice in accordance with simulated material and/or structuralcharacteristics of the first movement path in the virtualizedenvironment. The computing device modifies (816) at least a portion ofthe virtualized environment (e.g., marking the point cloud, saved as anordered point set) that corresponds to the locations of the one or moremarking inputs in the virtualized environment. The modified virtualizedenvironment replaces the virtualized environment as environment basisfor path planning for the robot within the first physical environmentthat is performed by the computing device in accordance with the firstset of preprogrammed path-planning instructions.

In some embodiments, to generate haptic feedback via the haptic-enabledinput device in accordance with simulated material and/or structuralcharacteristics of the first movement path in the virtualizedenvironment, the computing device performs the following operations. Thecomputing device, in accordance with a movement of the first pointerobject from a first location to a second location in the virtualizedenvironment, updates a proxy position for generating a force feedbackduring the movement of the first pointer object. Updating the proxyposition includes: in accordance with a determination that the firstlocation corresponds to an initial contact with the virtualizedenvironment along the first movement path, updating the proxy positionto locate a nearest surface point from the first location; and inaccordance with a determination that the first location is not theinitial contact with the virtualized environment along the firstmovement path, updating the proxy position to locate a nearest surfacepoint from an intersection point between a tangential constraint planecorresponding the first location and a normal line dropped to thetangential constraint plane from the first location. This is illustratedin FIG. 4 and accompanying descriptions. In some embodiments, updatingthe proxy position to locate a nearest surface point from the firstlocation utilizes intermediate results computed for ray-casting forvisual rendering of the one or more surfaces, and is completed inparallel to the ray-casting for visual rendering of the one or moresurfaces.

In some embodiments, modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment includes marking a path that is tobe followed by the robot (e.g., by marking 308), and the planned pathgenerated in accordance with the first set of preprogrammedpath-planning instructions is the marked path (e.g., as illustrated inFIG. 7B(b)).

In some embodiments, modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment includes marking a boundary thatprevents crossing by the robot, and the planned path generated inaccordance with the first set of preprogrammed path-planninginstructions does not cross the boundary.

In some embodiments, modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment includes placing one or morevirtual obstacles (e.g., obstacle 718-1) that are avoided by the robot,and the planned path generated in accordance with the first set ofpreprogrammed path-planning instructions precludes contact with the oneor more virtual obstacles (e.g., as illustrated in FIG. 7B(c)).

In some embodiments, modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment includes placing one or morevirtual passages (e.g., 718-3) at locations corresponding to physicalobjects in the first physical environment, wherein a path generated inaccordance with the first set of preprogrammed path-planninginstructions when the one or more virtual passages are not presentprecludes contact with the physical objects by the robot, and theplanned path generated in accordance with the first set of preprogramedpath-planning instructions when the one or more virtual passages arepresent does not preclude contact with the physical objects by the robotduring navigation (e.g., as illustrated in FIG. 7B(e)).

In some embodiments, modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment includes segmenting thevirtualized environment from the locations of the one or more markinginputs in the virtualized environment, and marking one or more physicalobjects corresponding to result of the segmenting as targets for thenavigation in accordance with the first set of preprogrammedpath-planning instructions (e.g., as illustrated in FIG. 7A, box 704 ismarked as a target). In some embodiments, the computing device providesa high-level instruction regarding manipulating the one or more physicalobjects in accordance with a second set of preprogrammedobject-manipulation instructions, wherein the robot manipulates the oneor more physical objects in accordance with the high-level instructionsand locally-stored task-execution instructions. For example, thehigh-level instruction is “rotate the box”, and the robot rotates thebox in accordance with locally-stored task-execution instructions thatincludes instructions for approaching the box, identifying surfaces ofthe box, identifying firm locations on the surfaces of the box forpushing the box, and pushing the identified locations to rotate the boxto a desired amount.

It should be understood that the particular order in which theoperations in method 800 have been described is merely an example and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 900 and 1000) are also applicable in an analogous manner tomethod 800 described above with respect to FIG. 8. For brevity, thesedetails are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors orapplication specific chips.

FIG. 9 is a flowchart of a method 900 of performing interactive objectsegmentation from streaming surfaces, in accordance with someembodiments. The method 900 of performing interactive objectsegmentation from streaming surfaces is performed (902) at a computingdevice having one or more processors and memory. The computing device iscommunicably coupled, via a network, to a set of sensors and a robotcollocated in a first physical environment. The computing devicecommunicably coupled to a robot and is configured to generate a plannedpath for the robot in accordance with a first set of preprogrammedpath-planning instructions. The robot is configured to navigate within aphysical environment in accordance with the planned path received fromthe computing device and locally-stored path-execution instructions. Insome embodiments, the computing device is a server that is collocatedwith the robot (e.g., using the same local area network or in the sameroom). In some embodiments, the computing device is a server that isremotely located from the robot, and communicates with the robot over awide area network, and may experience network delays when communicatingwith the robot over the network. In some embodiments, the computingdevice is a computer that is collocated with a human operator thatfacilitates remote interaction between the user and the robot.

In method 900, the computing device receives (904) an environment datastream from the set of sensors, the environment data stream includingcorrelated image and depth data capturing the first physicalenvironment. The computing device displays (906), via a displaygeneration component (e.g., display 112), a virtualized representationof the first physical environment, in a user interface region (e.g.,user interface region 702 in FIGS. 7A-7C), wherein the virtualizedrepresentation of the first physical environment is generated andupdated in accordance with the environment data stream. While displayingthe virtualized representation of the first physical environment, thecomputing device receives (908) a marking input from a haptic-enabledinput device. A position of the marking input in the virtualizedrepresentation of the first physical environment is determined inaccordance with the marking input and is constrained by the a firstvirtualized surface in the virtualized representation of the firstphysical environment. In response to receiving the marking input (910),the computing device performs (912) object segmentation from theposition of the marking input on the correlated image and depth data,including, for a respective time-step in updating the virtualizedrepresentation of the first physical environment in accordance with theenvironment data stream: clustering (912) pixels from a first positionin the image data that corresponds to the position of the marking inputto form a first cluster; segmenting out a first labeled object based onthe first cluster and the depth data corresponding to the first cluster;fusing the first labeled object and the depth data into a firstTruncated Signed Distance Function (TSDF) representation of the firstphysical environment corresponding to the respective time-step; andobtaining a first segmented point cloud from the first TSDFrepresentation fused with the first labeled object and the depth datacorresponding to the first cluster. This is illustrated in FIG. 6 andaccompanying descriptions.

In some embodiments, providing the first segmented point cloud includingthe first labeled object as input for path planning and/or taskexecution to be performed by a robot with respect to a first physicalobject corresponding to the first labeled object, in accordance withpreprogrammed instructions.

In some embodiments, the computing device generates haptic feedback viathe haptic-enabled input device in response to the marking input,including: in accordance with a determination that the location of themarking input corresponds to an initial contact with the virtualizedenvironment, updating a proxy position to locate a nearest surface pointfrom the location of the marking input; and in accordance with adetermination that the location of the marking input is a successivelocation of contact with the virtualized environment along a movementpath, updating the proxy position to locate a nearest surface point froman intersection point between a tangential constraint planecorresponding the first location and a normal line dropped to thetangential constraint plane from the first location. This is illustratedin FIG. 4 and accompanying descriptions. In some embodiments, updatingthe proxy position to locate a nearest surface point from the firstlocation is performed using the first TSDF representation of the firstphysical environment corresponding to the respective time-step.

In some embodiments, the computing device marks the first labeled objectas an obstacle in accordance with the marking input, wherein the robotavoids the first labeled object during navigation on a path generated inaccordance with the first set of preprogrammed instructions (e.g., asillustrated in FIG. 7B(c)).

In some embodiments, the computing device marks the first labeled objectas a target in accordance with the marking input, wherein the robotapproaches the first labeled object during navigation on a pathgenerated in accordance with a first set of preprogrammed path-planninginstructions (e.g., as illustrated in FIGS. 7B(a)-7B(f)).

In some embodiments, the computing device marks the first labeled objectas a passage in accordance with the marking input, wherein the robotavoids the first labeled object during navigation on a path generated inaccordance with a first set of preprogrammed path-planning instructionswhen the first labeled object is not marked as a passage, and the robotwill not avoid the first labeled object during navigation on a pathgenerated in accordance with the first set of preprogrammedpath-planning instructions when the first labeled object is marked as apassage (e.g., as illustrated in FIG. 7B(e)).

In some embodiments, the computing device marks the first labeled objectas a subject of manipulation in accordance with the marking input. Thecomputing device provides a high-level instruction regardingmanipulating the first labeled object to the robot, wherein the robotmanipulates a physical object corresponding to the first labeled objectthe high-level instructions generated in accordance with a second set ofpre-programmed object-manipulation instructions.

It should be understood that the particular order in which theoperations in method 900 have been described is merely an example and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 800 and 1000) are also applicable in an analogous manner tomethod 900 described above with respect to FIG. 8. For brevity, thesedetails are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors orapplication specific chips.

FIG. 10 is a flowchart of a method of providing mixed-initiative roboticcontrol, in accordance with some embodiments. The method 1000 ofproviding mixed-initiative robotic control is performed (1002) at acomputing device having one or more processors and memory. The computingdevice (e.g., server 110) communicably coupled to a robot and isconfigured to generate a planned path for the robot in accordance with afirst set of preprogrammed path-planning instructions. The robot (e.g.,robot 102) is configured to navigate within a physical environment inaccordance with the planned path received from the computing device andlocally-stored path-execution instructions.

In method 1000, the computing device displays (1004) a control userinterface (e.g., control user interface 700) via a display generationcomponent coupled to the computing device, including displaying avirtualized environment (e.g., in region 702) corresponding to a firstphysical environment currently surrounding the robot (e.g., robot 102),wherein the virtualized environment is generated and updated inaccordance with streaming environment data received from a first set ofsensors (e.g., 3D camera 106) collocated with the robot. Whiledisplaying the virtualized environment, the computing device detects(1006) a first user input inserting a first virtual object at a firstlocation in the virtualized environment (e.g., via the control userinterface 700). In response to detecting the first user input, thecomputing device modifies (1008) the first virtualized environment inaccordance with the insertion of the first virtual object at the firstlocation, wherein the first virtual object at the first location causesthe robot to execute a first navigation path in the physical environmentthat is generated in accordance with the first set of pre-programmedpath-planning instructions. While displaying the first virtual object atthe first location in the virtualized environment and while the robot isexecuting the first navigation path in the physical environment, thecomputing device detects (1010) a second user input, including detectinga first movement input directed to the first virtual object via ahaptic-enabled input device. In response to detecting the second userinput, the computing device moves (1012) the first virtual object alonga first movement path to a second location in the virtualizedenvironment in accordance with the first movement input, wherein thefirst movement path is constrained by one or more simulated surfaces inthe virtualized environment, wherein the first virtual object at thesecond location causes the robot to execute a modified navigation pathin the physical environment that is generated in accordance with thefirst set of pre-programmed path-planning instructions. This isillustrated in FIGS. 7B-7C and accompanying descriptions.

In some embodiments, during movement of the first virtual object alongthe first movement path to the second location in the virtualizedenvironment, the computing device provides haptic feedback via thehaptic-enabled input device (e.g., input device 114) in accordance withsimulated friction forces between the first virtual object and asimulated surface at a current location of the first virtual object.

In some embodiments, the first virtual object has a predefined size andshape. During movement of the first virtual object along the firstmovement path to the second location in the virtualized environment, thecomputing device detects that the first virtual object has come incontact with a first simulated surface in a current direction ofmovement of the first virtual object. In response to detecting that thefirst virtual object has come in contact with the first simulatedsurface in the current direction of movement of the first virtualobject, the computing device provides haptic feedback via thehaptic-enabled input device in accordance with simulated reactive forcesfrom the first simulated surface to the first virtual object. This isillustrated in FIG. 4, for example.

In some embodiments, the first virtual object has a predefined size andshape. During movement of the first virtual object along the firstmovement path to the second location in the virtualized environment, thecomputing device detects that the first virtual object has come incontact with a second virtual object that was previously placed in thesimulated environment in the current direction of movement of the firstvirtual object. In response to detecting that the first virtual objecthas come in contact with the second virtual object in the currentdirection of movement of the first virtual object, the computing deviceprovides haptic feedback via the haptic-enabled input device inaccordance with simulated reactive forces from the second virtual objectto the first virtual object. This is illustrated in FIG. 5, for example.

In some embodiments, the first virtual object is marked as an obstacleand the robot avoids the first virtual object during navigation on apath generated in accordance with the first set of pre-programmedinstructions (e.g., as illustrated in FIG. 7B(c)).

In some embodiments, the first virtual object is marked as a target andthe robot approaches the first labeled object during navigation on apath generated in accordance with the first set of pre-programmedinstructions (e.g., as illustrated in FIGS. 7B(a)-(f)).

In some embodiments, the first virtual object is marked as a passage,the presence of the first virtual object overrides presence of aphysical obstacle during the robot's navigation on a path generated inaccordance with the first set of preprogrammed path-planninginstructions (e.g., as illustrated in FIG. 7B(e)).

In some embodiments, to detect the first user input inserting the firstvirtual object at the first location in the virtualized environment, thecomputing device performs the following operations: detecting a firstinput directed to a first position in the virtualized environment thatcorresponds to a first simulated surface; in response to detecting thefirst input directed to the first position, providing first hapticfeedback via the haptic-enabled device indicating that the firstposition is on the first simulated surface; detecting a second inputmarking the first position in the virtualized environment as part of afirst boundary of the first virtual object; detecting a third inputdirected to a second position in the virtualized environment thatcorresponds to a second simulated surface; in response to detecting thesecond input directed to the second position, providing second hapticfeedback via the haptic-enabled device indicating that the secondposition is on the second simulated surface; detecting a fourth inputmarking the second position in the virtualized environment as part of asecond boundary of the first virtual object; and generating the firstvirtual object in accordance with the first position and the secondposition.

In some embodiments, the computing device marks the first labeled objectas a subject of manipulation in accordance with the marking input. Thecomputing device provides a high-level instruction regardingmanipulating the first labeled object to the robot, wherein the robotmanipulates a physical object corresponding to the first labeled objectthe high-level instructions generated in accordance with a second set ofpre-programmed object-manipulation instructions.

It should be understood that the particular order in which theoperations in method 1000 have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 800 and 900) are also applicable in an analogous manner tomethod 1000 described above with respect to FIG. 8. For brevity, thesedetails are not repeated here.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors orapplication specific chips.

FIG. 11 is a block diagram illustrating an exemplary robot 102 inaccordance with some embodiments.

The robot 102 includes one or more processing units (CPUs) 1102, one ormore network interfaces 1104 (e.g., including the I/O interface toserver 110), memory 1106, and one or more communication buses 1108 forinterconnecting these components (sometimes called a chipset). Thememory 1106 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 1106, optionally, includes one or morestorage devices remotely located from the one or more processing units1102. The memory 1106, or alternatively the non-volatile memory withinthe memory 1106, includes a non-transitory computer readable storagemedium. In some implementations, the memory 1106, or the non-transitorycomputer readable storage medium of the memory 1106, stores thefollowing programs, modules, and data structures, or a subset orsuperset thereof:

-   -   Operating system 1116 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 1118 for connecting the robot 102        to other computing devices;    -   Presentation module 1120 for enabling presentation of        information at the robot 102 via the one or more output devices        1112 (e.g., displays, speakers, etc.) associated with the user        interface 1110;    -   Input processing module 1122 for detecting one or more user        inputs or interactions from one of the one or more input devices        1114 and interpreting the detected input or interaction;    -   Robot Control Unit 1124 for controlling functions of the robot        102, including camera control unit 1113 for controlling RGBD        camera 106 of the robot 102, robot function control unit 1117        for controlling the robot operation unit 1126 of the robot, path        execution unit 1119 for executing a path in accordance with a        high level instructions of a path planner, and other units for        implementing the functions of the robot 102 as described herein.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 1106, optionally, stores a subset of the modulesand data structures identified above. Furthermore, the memory 1106,optionally, stores additional modules and data structures not describedabove.

In some implementations, at least some of the functions of the robot 102are performed by the server 110, and the corresponding sub-modules ofthese functions may be located within the server 110 rather than therobot 102. The robot 102 shown in FIG. 11 is merely illustrative, anddifferent configurations of the modules for implementing the functionsdescribed herein are possible in various implementations.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 1106, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 1106, optionally,stores additional modules and data structures not described above.

FIG. 12 is a block diagram illustrating an exemplary server 110 inaccordance with some implementations. The server system 110, typically,includes one or more processing units (CPUs) 1202, one or more networkinterfaces 1204 (e.g., including the I/O interface to one or more robots102 and the I/O interface to one or more user-side device 1300), memory1206, and one or more communication buses 1208 for interconnecting thesecomponents (sometimes called a chipset). The memory 1206 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM, or otherrandom access solid state memory devices; and, optionally, includesnon-volatile memory, such as one or more magnetic disk storage devices,one or more optical disk storage devices, one or more flash memorydevices, or one or more other non-volatile solid state storage devices.The memory 1206, optionally, includes one or more storage devicesremotely located from the one or more processing units 1202. The memory1206, or alternatively the non-volatile memory within the memory 1206,includes a non-transitory computer readable storage medium. In someimplementations, the memory 1206, or the non-transitory computerreadable storage medium of the memory 1206, stores the followingprograms, modules, and data structures, or a subset or superset thereof:

-   -   Operating system 1216 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 1218 for connecting the server 110        to other computing devices (e.g., the user-side device 1300 and        the robots 102 (including e.g., camera(s) 106));    -   Presentation module 1220 for enabling presentation of        information at the server 110 via the one or more output devices        1212 (e.g., displays, speakers, etc.) associated with the user        interface 1210;    -   Input processing module 1222 for detecting one or more user        inputs or interactions from one of the one or more input devices        1214 and interpreting the detected input or interaction;    -   Server-side modules 1226 for controlling functions of the server        110, including object segmentation module 1228 for performing        object segmentation in the virtualized environment, KinectFusion        module 1230 for generating and updating the virtualized        environment based on the image and depth data stream received        from the robot, Haptic rendering module 1232 for generating        haptic feedback based on the user's input provided via the        haptic enabled input device and based on the location of the        input in the virtualized environment, physics simulation module        for generating reaction and friction force rendering as well as        object interaction models in the virtualized environment, path        planning module 1236 for generating a planned path based on the        virtualized environment and haptics markings and virtual objects        present in the virtualized environment, controller module 1238        for controlling path execution by the robot, network latency        compensation module 1240 for adjusting path planning based on        network delays, and other modules 1242 for implementing other        functions of the server 110 as described herein.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 1206, optionally, stores a subset of the modulesand data structures identified above. Furthermore, the memory 1206,optionally, stores additional modules and data structures not describedabove.

In some implementations, at least some of the functions of the server110 are performed by the robot 110 or the user-side device 1300, and thecorresponding sub-modules of these functions may be located within theserver robot or user-side device rather than the server 110. The server110 shown in FIG. 12 is merely illustrative, and differentconfigurations of the modules for implementing the functions describedherein are possible in various implementations.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 1206, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 1206, optionally,stores additional modules and data structures not described above.

FIG. 13 is a block diagram illustrating an exemplary user-side device1300 in accordance with some implementations. The user-side device 1300,typically, includes one or more processing units (CPUs) 1302, one ormore network interfaces 1304 (e.g., including the I/O interface toserver 110), memory 1306, and one or more communication buses 1308 forinterconnecting these components (sometimes called a chipset). Thememory 1306 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 1306, optionally, includes one or morestorage devices remotely located from the one or more processing units1302. The memory 1306, or alternatively the non-volatile memory withinthe memory 1306, includes a non-transitory computer readable storagemedium. In some implementations, the memory 1306, or the non-transitorycomputer readable storage medium of the memory 1306, stores thefollowing programs, modules, and data structures, or a subset orsuperset thereof:

-   -   Operating system 1316 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 1318 for connecting the server 110        to other computing devices;    -   Presentation module 1320 for enabling presentation of        information at the user-side device 1300 via the one or more        output devices 1312 (e.g., displays 112, speakers,        haptic-enabled input device 114, etc.) associated with the user        interface 1310;    -   Input processing module 1322 for detecting one or more user        inputs or interactions from one of the one or more input devices        1314 (e.g., haptic-enabled input device 114) and interpreting        the detected input or interaction;    -   Applications 1324 for implementing various user-level functions,        such as word processing, drawing, etc.    -   User-side modules 1326 for controlling functions of the        user-side devices, including user layer module 1328 and other        modules 1330 for implementing other functions of the user-side        device as described herein.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 1306, optionally, stores a subset of the modulesand data structures identified above. Furthermore, the memory 1306,optionally, stores additional modules and data structures not describedabove.

In some implementations, at least some of the functions of the server110 are performed by the user-side device 1300, and the correspondingsub-modules of these functions may be located within the server ratherthan the user-side device 1300. The user-side device 1300 shown in FIG.13 is merely illustrative, and different configurations of the modulesfor implementing the functions described herein are possible in variousimplementations.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 1306, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 1306, optionally,stores additional modules and data structures not described above.

What is claimed is:
 1. A method of providing mixed-initiative roboticcontrol, including: at a computing device having one or more processorsand memory, wherein the computing device is communicably coupled to arobot and is configured to generate a planned path for the robot inaccordance with a first set of preprogrammed path-planning instructions,and the robot is configured to navigate within a physical environment inaccordance with the planned path received from the computing device andlocally-stored path-execution instructions; displaying a control userinterface via a display generation component coupled to the computingdevice, including displaying a virtual pointer object within avirtualized environment corresponding to the physical environmentcurrently surrounding the robot, wherein the virtualized environment isgenerated and updated in accordance with streaming environment datareceived from a first set of sensors collocated with the robot; whiledisplaying the virtual pointer object in the virtualized environment,detecting a first user input via a haptic-enabled input device,including detecting a first movement input and one or more markinginputs at one or more locations along a movement path of the virtualpointer object in the virtualized environment; in response to detectingthe first user input: moving the virtual pointer object along themovement path in the virtualized environment in accordance with thefirst movement input, wherein the movement path is constrained by one ormore simulated surfaces in the virtualized environment; generatinghaptic feedback via the haptic-enabled input device in accordance withsimulated material and/or structural characteristics of the movementpath in the virtualized environment; and modifying at least a portion ofthe virtualized environment that corresponds to the locations of the oneor more marking inputs in the virtualized environment, wherein themodified virtualized environment replaces the virtualized environment asenvironment basis for path planning for the robot within the physicalenvironment that is performed by the computing device in accordance withthe first set of preprogrammed path-planning instructions.
 2. The methodof claim 1, wherein generating haptic feedback via the haptic-enabledinput device in accordance with simulated material and/or structuralcharacteristics of the movement path in the virtualized environmentincludes: in accordance with a movement of the pointer object from afirst location to a second location in the virtualized environment,updating a proxy position for generating a force feedback during themovement of the pointer object, including: in accordance with adetermination that the first location corresponds to an initial contactwith the virtualized environment along the movement path, updating theproxy position to locate a nearest surface point from the firstlocation; and in accordance with a determination that the first locationis not the initial contact with the virtualized environment along themovement path, updating the proxy position to locate a nearest surfacepoint from an intersection point between a tangential constraint planecorresponding the first location and a normal line dropped to thetangential constraint plane from the first location.
 3. The method ofclaim 1, wherein modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment includes marking a path that is tobe followed by the robot, and the planned path generated in accordancewith the first set of preprogrammed path-planning instructions is themarked path.
 4. The method of claim 1, wherein modifying at least aportion of the virtualized environment that corresponds to the locationsof the one or more marking inputs in the virtualized environmentincludes marking a boundary that prevents crossing by the robot, and theplanned path generated in accordance with the first set of preprogrammedpath-planning instructions does not cross the boundary.
 5. The method ofclaim 1, wherein modifying at least a portion of the virtualizedenvironment that corresponds to the locations of the one or more markinginputs in the virtualized environment includes placing one or morevirtual obstacles that are avoided by the robot, and the planned pathgenerated in accordance with the first set of preprogrammedpath-planning instructions precludes contact with the one or morevirtual obstacles by the robot.
 6. The method of claim 1, whereinmodifying at least a portion of the virtualized environment thatcorresponds to the locations of the one or more marking inputs in thevirtualized environment includes placing one or more virtual passages atlocations corresponding to physical objects in the physical environment,wherein the planned path generated in accordance with the first set ofpreprogrammed path-planning instructions when the one or more virtualpassages are not present precludes contact with the physical objects bythe robot, and the planned path generated in accordance with the firstset of preprogramed path-planning instructions when the one or morevirtual passages are present does not preclude contact with the physicalobjects by the robot during navigation.
 7. The method of claim 1,wherein modifying at least a portion of the virtualized environment thatcorresponds to the locations of the one or more marking inputs in thevirtualized environment includes segmenting the virtualized environmentfrom the locations of the one or more marking inputs in the virtualizedenvironment, and marking one or more physical objects corresponding to aresult of the segmenting as targets for the navigation in accordancewith the first set of preprogrammed path-planning instructions.
 8. Acomputing device for providing mixed-initiative robotic control,comprising: one or more processors; and memory storing instructions,wherein: the computing device is communicably coupled to a robot and isconfigured to generate a planned path for the robot in accordance with afirst set of preprogrammed path-planning instructions, the robot isconfigured to navigate within a physical environment in accordance withthe planned path received from the computing device and locally-storedpath-execution instructions, and the instructions, when executed by theone or more processors, cause the processors to perform operationscomprising: displaying a control user interface via a display generationcomponent coupled to the computing device, including displaying avirtual pointer object within a virtualized environment corresponding tothe physical environment currently surrounding the robot, wherein thevirtualized environment is generated and updated in accordance withstreaming environment data received from a first set of sensorscollocated with the robot; while displaying the virtual pointer objectin the virtualized environment, detecting a first user input via ahaptic-enabled input device, including detecting a first movement inputand one or more marking inputs at one or more locations along a movementpath of the virtual pointer object in the virtualized environment; andin response to detecting the first user input: moving the virtualpointer object along the movement path in the virtualized environment inaccordance with the first movement input, wherein the movement path isconstrained by one or more simulated surfaces in the virtualizedenvironment; generating haptic feedback via the haptic-enabled inputdevice in accordance with simulated material and/or structuralcharacteristics of the movement path in the virtualized environment; andmodifying at least a portion of the virtualized environment thatcorresponds to the locations of the one or more marking inputs in thevirtualized environment, wherein the modified virtualized environmentreplaces the virtualized environment as environment basis forpath-planning for the robot within the physical environment that isperformed by the computing device in accordance with the first set ofpreprogrammed path-planning instructions.
 9. The computing device ofclaim 8, wherein generating haptic feedback via the haptic enabled inputdevice in accordance with simulated material and/or structuralcharacteristics of the movement path in the virtualized environmentincludes: in accordance with a movement of the pointer object from afirst location to a second location in the virtualized environment,updating a proxy position for generating a force feedback during themovement of the pointer object, including: in accordance with adetermination that the first location corresponds to an initial contactwith the virtualized environment along the movement path, updating theproxy position to locate a nearest surface point from the firstlocation; and in accordance with a determination that the first locationis not the initial contact with the virtualized environment along thefirst movement path, updating the proxy position to locate a nearestsurface point from an intersection point between a tangential constraintplane corresponding the first location and a normal line dropped to thetangential constraint plane from the first location.
 10. The computingdevice of claim 8, wherein modifying at least a portion of thevirtualized environment that corresponds to the locations of the one ormore marking inputs in the virtualized environment includes marking apath that is to be followed by the robot, and the planned path generatedin accordance with the first set of preprogrammed path-planninginstructions is the marked path.
 11. The computing device of claim 8,wherein modifying at least a portion of the virtualized environment thatcorresponds to the locations of the one or more marking inputs in thevirtualized environment includes marking a boundary that preventscrossing by the robot, and the planned path generated in accordance withthe first set of preprogrammed path-planning instructions does not crossthe boundary.
 12. The computing device of claim 8, wherein modifying atleast a portion of the virtualized environment that corresponds to thelocations of the one or more marking inputs in the virtualizedenvironment includes placing one or more virtual obstacles that areavoided by the robot, and the planned path generated in accordance withthe first set of preprogrammed path-planning instructions precludescontact with the one or more virtual obstacles by the robot.
 13. Thecomputing device of claim 8, wherein modifying at least a portion of thevirtualized environment that corresponds to the locations of the one ormore marking inputs in the virtualized environment includes placing oneor more virtual passages at locations corresponding to physical objectsin the physical environment, wherein a path generated in accordance withthe first set of preprogrammed path-planning instructions when the oneor more virtual passages are not present precludes contact with thephysical objects by the robot, and the planned path generated inaccordance with the first set of preprogramed path-planning instructionswhen the one or more virtual passages are present does not precludecontact with the physical objects by the robot during navigation. 14.The computing device of claim 8, wherein modifying at least a portion ofthe virtualized environment that corresponds to the locations of the oneor more marking inputs in the virtualized environment includessegmenting the virtualized environment from the locations of the one ormore marking inputs in the virtualized environment, and marking one ormore physical objects corresponding to a result of the segmenting astargets for the navigation in accordance with the first set ofpreprogrammed path-planning instructions.
 15. A computer-readablestorage medium for providing mixed-initiative robotic control, whereinthe computer-readable storage medium stores instructions, theinstructions, when executed by one or more processors of a computingdevice, cause the computing device to perform operations, wherein: thecomputing device is communicably coupled to a robot and is configured togenerate a planned path for the robot in accordance with a first set ofpreprogrammed path-planning instructions, the robot configured tonavigate within a physical environment in accordance with the plannedpath received from the computing device and locally-storedpath-execution instructions, and the operations include: displaying acontrol user interface via a display generation component coupled to thecomputing device, including displaying a virtual pointer object within avirtualized environment corresponding to the physical environmentcurrently surrounding the robot, wherein the virtualized environment isgenerated and updated in accordance with streaming environment datareceived from a first set of sensors collocated with the robot; whiledisplaying the virtual pointer object in the virtualized environment,detecting a first user input via a haptic-enabled input device,including detecting a first movement input and one or more markinginputs at one or more locations along a movement path of the virtualpointer object in the virtualized environment; and in response todetecting the first user input: moving the virtual pointer object alongthe movement path in the virtualized environment in accordance with thefirst movement input, wherein the movement path is constrained by one ormore simulated surfaces in the virtualized environment; generatinghaptic feedback via the haptic-enabled input device in accordance withsimulated material and/or structural characteristics of the movementpath in the virtualized environment; and modifying at least a portion ofthe virtualized environment that corresponds to the locations of the oneor more marking inputs in the virtualized environment, wherein themodified virtualized environment replaces the virtualized environment asenvironment basis for path-planning for the robot within the physicalenvironment that is performed by the computing device in accordance withthe first set of preprogrammed path-planning instructions. environmentas environment basis for path planning for the robot within the firstphysical environment that is performed by the computing device inaccordance with the first set of preprogrammed path-planninginstructions.
 16. The computer-readable storage medium of claim 15,wherein generating haptic feedback via the haptic-enabled input devicein accordance with simulated material and/or structural characteristicsof the movement path in the virtualized environment includes: inaccordance with a movement of the pointer object from a first locationto a second location in the virtualized environment, updating a proxyposition for generating a force feedback during the movement of thepointer object, including: in accordance with a determination that thefirst location corresponds to an initial contact with the virtualizedenvironment along the movement path, updating the proxy position tolocate a nearest surface point from the first location; and inaccordance with a determination that the first location is not theinitial contact with the virtualized environment along the movementpath, updating the proxy position to locate a nearest surface point froman intersection point between a tangential constraint planecorresponding the first location and a normal line dropped to thetangential constraint plane from the first location.
 17. Thecomputer-readable storage medium of claim 15, wherein modifying at leasta portion of the virtualized environment that corresponds to thelocations of the one or more marking inputs in the virtualizedenvironment includes marking a path that is to be followed by the robot,and the planned path generated in accordance with the first set ofpreprogrammed path-planning instructions is the marked path.
 18. Thecomputer-readable storage medium of claim 15, wherein modifying at leasta portion of the virtualized environment that corresponds to thelocations of the one or more marking inputs in the virtualizedenvironment includes marking a boundary that prevents crossing by therobot, and the planned path generated in accordance with the first setof preprogrammed path-planning instructions does not cross the boundary.19. The computer-readable storage medium of claim 15, wherein modifyingat least a portion of the virtualized environment that corresponds tothe locations of the one or more marking inputs in the virtualizedenvironment includes placing one or more virtual obstacles that areavoided by the robot, and the planned path generated in accordance withthe first set of preprogrammed path-planning instructions precludescontact with the one or more virtual obstacles by the robot.
 20. Thecomputer-readable storage medium of claim 15, wherein modifying at leasta portion of the virtualized environment that corresponds to thelocations of the one or more marking inputs in the virtualizedenvironment includes placing one or more virtual passages at locationscorresponding to physical objects in the physical environment, wherein apath generated in accordance with the first set of preprogrammedpath-planning instructions when the one or more virtual passages are notpresent precludes contact with the physical objects by the robot, andthe planned path generated in accordance with the first set ofpreprogramed path-planning instructions when the one or more virtualpassages are present does not preclude contact with the physical objectsby the robot during navigation.